[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704375975685-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 17:52:56 dist-test-slave-53nh krb5kdc[399](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 17:52:56 dist-test-slave-53nh krb5kdc[399](info): set up 2 sockets
Sep 29 17:52:56 dist-test-slave-53nh krb5kdc[399](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260929 17:52:56.961064 392 mini_kdc.cc:121] Time spent starting KDC: real 0.948s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704375975685-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704375975685-392-0/krb5kdc/test-user.keytab.
Sep 29 17:52:57 dist-test-slave-53nh krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704377, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T17:52:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T17:52:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704375975685-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704375975685-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 17:52:57.078517 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskup3Soe/build/debug/bin/kudu
/tmp/dist-test-taskup3Soe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704375975685-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704375975685-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704375975685-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704375975685-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38289
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704375975685-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38289
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704375975685-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704375975685-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704375975685-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:52:57.183949 412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:52:57.184203 412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:52:57.184265 412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:52:57.188148 412 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 17:52:57.188200 412 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:52:57.188234 412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:52:57.188257 412 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 17:52:57.188285 412 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 17:52:57.193426 412 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704375975685-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704375975685-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38289
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704375975685-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38289
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704375975685-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.412
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704375975685-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:52:57.194387 412 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:52:57.195365 412 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:52:57.203974 417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:52:57.204000 412 server_base.cc:1034] running on GCE node
W20260929 17:52:57.203925 418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:52:57.204102 420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:52:57.204737 412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:52:57.205945 412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:52:57.207094 412 hybrid_clock.cc:648] HybridClock initialized: now 1790704377207047 us; error 59 us; skew 500 ppm
Sep 29 17:52:57 dist-test-slave-53nh krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704377, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:52:57.209684 412 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 17:52:57.210515 412 webserver.cc:460] Webserver started at http://127.0.98.62:35395/ using document root <none> and password file <none>
I20260929 17:52:57.211021 412 fs_manager.cc:359] Metadata directory not provided
I20260929 17:52:57.211078 412 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:52:57.211267 412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:52:57.213254 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704375975685-392-0/minicluster-data/master-0/data/instance:
uuid: "dec1e1dd039a4344a8610ac85a34e3da"
format_stamp: "Formatted at 2026-09-29 17:52:57 on dist-test-slave-53nh"
server_key: "9a819dcf333460d8af9f0a509a36d0a0"
server_key_iv: "9f24d44906e67406d9e8a677a53c5694"
server_key_version: "encryptionkey@0"
I20260929 17:52:57.213783 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704375975685-392-0/minicluster-data/master-0/wal/instance:
uuid: "dec1e1dd039a4344a8610ac85a34e3da"
format_stamp: "Formatted at 2026-09-29 17:52:57 on dist-test-slave-53nh"
server_key: "9a819dcf333460d8af9f0a509a36d0a0"
server_key_iv: "9f24d44906e67406d9e8a677a53c5694"
server_key_version: "encryptionkey@0"
I20260929 17:52:57.217214 412 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260929 17:52:57.219365 427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:52:57.220547 412 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 17:52:57.220678 412 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704375975685-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704375975685-392-0/minicluster-data/master-0/wal
uuid: "dec1e1dd039a4344a8610ac85a34e3da"
format_stamp: "Formatted at 2026-09-29 17:52:57 on dist-test-slave-53nh"
server_key: "9a819dcf333460d8af9f0a509a36d0a0"
server_key_iv: "9f24d44906e67406d9e8a677a53c5694"
server_key_version: "encryptionkey@0"
I20260929 17:52:57.220769 412 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704375975685-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704375975685-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704375975685-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:52:57.239310 412 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:52:57.242409 412 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:52:57.242635 412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:52:57.249881 412 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38289
I20260929 17:52:57.249897 479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38289 every 8 connection(s)
I20260929 17:52:57.250854 412 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704375975685-392-0/minicluster-data/master-0/data/info.pb
I20260929 17:52:57.253566 480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:52:57.254405 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskup3Soe/build/debug/bin/kudu as pid 412
I20260929 17:52:57.254534 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704375975685-392-0/minicluster-data/master-0/wal/instance
I20260929 17:52:57.255752 392 external_mini_cluster.cc:1428] Setting key b0abb7e5191e4af285b5207ab01cfa8a
I20260929 17:52:57.258884 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dec1e1dd039a4344a8610ac85a34e3da: Bootstrap starting.
I20260929 17:52:57.261032 480 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dec1e1dd039a4344a8610ac85a34e3da: Neither blocks nor log segments found. Creating new log.
I20260929 17:52:57.261708 480 log.cc:824] T 00000000000000000000000000000000 P dec1e1dd039a4344a8610ac85a34e3da: Log is configured to *not* fsync() on all Append() calls
I20260929 17:52:57.263625 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dec1e1dd039a4344a8610ac85a34e3da: No bootstrap required, opened a new log
I20260929 17:52:57.267361 480 raft_consensus.cc:348] T 00000000000000000000000000000000 P dec1e1dd039a4344a8610ac85a34e3da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dec1e1dd039a4344a8610ac85a34e3da" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38289 } }
I20260929 17:52:57.267611 480 raft_consensus.cc:374] T 00000000000000000000000000000000 P dec1e1dd039a4344a8610ac85a34e3da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:52:57.267680 480 raft_consensus.cc:729] T 00000000000000000000000000000000 P dec1e1dd039a4344a8610ac85a34e3da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dec1e1dd039a4344a8610ac85a34e3da, State: Initialized, Role: FOLLOWER
I20260929 17:52:57.268076 480 consensus_queue.cc:260] T 00000000000000000000000000000000 P dec1e1dd039a4344a8610ac85a34e3da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dec1e1dd039a4344a8610ac85a34e3da" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38289 } }
I20260929 17:52:57.268198 480 raft_consensus.cc:388] T 00000000000000000000000000000000 P dec1e1dd039a4344a8610ac85a34e3da [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 17:52:57.268256 480 raft_consensus.cc:482] T 00000000000000000000000000000000 P dec1e1dd039a4344a8610ac85a34e3da [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 17:52:57.268353 480 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dec1e1dd039a4344a8610ac85a34e3da [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:52:57.269299 480 raft_consensus.cc:504] T 00000000000000000000000000000000 P dec1e1dd039a4344a8610ac85a34e3da [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dec1e1dd039a4344a8610ac85a34e3da" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38289 } }
I20260929 17:52:57.269645 480 leader_election.cc:302] T 00000000000000000000000000000000 P dec1e1dd039a4344a8610ac85a34e3da [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dec1e1dd039a4344a8610ac85a34e3da; no voters:
I20260929 17:52:57.269972 480 leader_election.cc:288] T 00000000000000000000000000000000 P dec1e1dd039a4344a8610ac85a34e3da [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 17:52:57.270044 485 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dec1e1dd039a4344a8610ac85a34e3da [term 1 FOLLOWER]: Leader election won for term 1
I20260929 17:52:57.270211 485 raft_consensus.cc:686] T 00000000000000000000000000000000 P dec1e1dd039a4344a8610ac85a34e3da [term 1 LEADER]: Becoming Leader. State: Replica: dec1e1dd039a4344a8610ac85a34e3da, State: Running, Role: LEADER
I20260929 17:52:57.270498 485 consensus_queue.cc:237] T 00000000000000000000000000000000 P dec1e1dd039a4344a8610ac85a34e3da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dec1e1dd039a4344a8610ac85a34e3da" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38289 } }
I20260929 17:52:57.271194 480 sys_catalog.cc:564] T 00000000000000000000000000000000 P dec1e1dd039a4344a8610ac85a34e3da [sys.catalog]: configured and running, proceeding with master startup.
I20260929 17:52:57.271664 487 sys_catalog.cc:455] T 00000000000000000000000000000000 P dec1e1dd039a4344a8610ac85a34e3da [sys.catalog]: SysCatalogTable state changed. Reason: New leader dec1e1dd039a4344a8610ac85a34e3da. Latest consensus state: current_term: 1 leader_uuid: "dec1e1dd039a4344a8610ac85a34e3da" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dec1e1dd039a4344a8610ac85a34e3da" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38289 } } }
I20260929 17:52:57.271865 487 sys_catalog.cc:458] T 00000000000000000000000000000000 P dec1e1dd039a4344a8610ac85a34e3da [sys.catalog]: This master's current role is: LEADER
I20260929 17:52:57.272100 486 sys_catalog.cc:455] T 00000000000000000000000000000000 P dec1e1dd039a4344a8610ac85a34e3da [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dec1e1dd039a4344a8610ac85a34e3da" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dec1e1dd039a4344a8610ac85a34e3da" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38289 } } }
I20260929 17:52:57.272212 486 sys_catalog.cc:458] T 00000000000000000000000000000000 P dec1e1dd039a4344a8610ac85a34e3da [sys.catalog]: This master's current role is: LEADER
I20260929 17:52:57.272403 492 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Sep 29 17:52:57 dist-test-slave-53nh krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704377, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 17:52:57.275633 492 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 17:52:57.280510 482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:52:57.258699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37472 (local address 127.0.98.62:38289)
0929 17:52:57.259201 (+ 502us) server_negotiation.cc:206] Beginning negotiation
0929 17:52:57.259219 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 17:52:57.260182 (+ 963us) server_negotiation.cc:407] Connection header received
0929 17:52:57.263340 (+ 3158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:52:57.263361 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:52:57.263729 (+ 368us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:52:57.264092 (+ 363us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:52:57.266068 (+ 1976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:52:57.267556 (+ 1488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:52:57.268869 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:52:57.269548 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:52:57.269698 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:52:57.274386 (+ 4688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:52:57.274393 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:52:57.274408 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:52:57.274441 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:52:57.276452 (+ 2011us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:52:57.277138 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:52:57.277144 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:52:57.277150 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:52:57.277303 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:52:57.277690 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:52:57.277694 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:52:57.277695 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:52:57.278186 (+ 491us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:52:57.278386 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0929 17:52:57.279036 (+ 650us) server_negotiation.cc:299] Negotiation successful
0929 17:52:57.279341 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":351,"thread_start_us":136,"threads_started":1}
I20260929 17:52:57.282274 492 catalog_manager.cc:1347] Generated new cluster ID: 3d30aa460e5c445c9df246b73730aac5
I20260929 17:52:57.282330 492 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 17:52:57.301023 492 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 17:52:57.302347 492 catalog_manager.cc:1504] Loading token signing keys...
I20260929 17:52:57.310827 492 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dec1e1dd039a4344a8610ac85a34e3da: Generated new TSK 0
I20260929 17:52:57.311618 492 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704375975685-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704375975685-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260929 17:52:57.344542 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskup3Soe/build/debug/bin/kudu
/tmp/dist-test-taskup3Soe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704375975685-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704375975685-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704375975685-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704375975685-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38289
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704375975685-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704375975685-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704375975685-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704375975685-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:52:57.450579 506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:52:57.450819 506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:52:57.450875 506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:52:57.454762 506 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:52:57.454815 506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:52:57.454891 506 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 17:52:57.459913 506 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704375975685-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704375975685-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704375975685-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704375975685-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38289
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.506
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704375975685-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:52:57.460903 506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:52:57.461925 506 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:52:57.470947 512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:52:57.470986 506 server_base.cc:1034] running on GCE node
W20260929 17:52:57.470952 514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:52:57.470966 511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:52:57.471565 506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:52:57.472168 506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:52:57.473315 506 hybrid_clock.cc:648] HybridClock initialized: now 1790704377473298 us; error 30 us; skew 500 ppm
Sep 29 17:52:57 dist-test-slave-53nh krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704377, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:52:57.475786 506 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260929 17:52:57.476567 506 webserver.cc:460] Webserver started at http://127.0.98.1:44421/ using document root <none> and password file <none>
I20260929 17:52:57.477113 506 fs_manager.cc:359] Metadata directory not provided
I20260929 17:52:57.477180 506 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:52:57.477370 506 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:52:57.479323 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704375975685-392-0/minicluster-data/ts-0/data/instance:
uuid: "1289dd7ce2bb4d99b099c5707cdf2fdc"
format_stamp: "Formatted at 2026-09-29 17:52:57 on dist-test-slave-53nh"
server_key: "6650c3d27fe515f741b3fb3726ee1f89"
server_key_iv: "4a90c43e9765c8a8683235092f52c969"
server_key_version: "encryptionkey@0"
I20260929 17:52:57.479851 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704375975685-392-0/minicluster-data/ts-0/wal/instance:
uuid: "1289dd7ce2bb4d99b099c5707cdf2fdc"
format_stamp: "Formatted at 2026-09-29 17:52:57 on dist-test-slave-53nh"
server_key: "6650c3d27fe515f741b3fb3726ee1f89"
server_key_iv: "4a90c43e9765c8a8683235092f52c969"
server_key_version: "encryptionkey@0"
I20260929 17:52:57.483317 506 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 17:52:57.485605 521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:52:57.486771 506 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 17:52:57.486872 506 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704375975685-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704375975685-392-0/minicluster-data/ts-0/wal
uuid: "1289dd7ce2bb4d99b099c5707cdf2fdc"
format_stamp: "Formatted at 2026-09-29 17:52:57 on dist-test-slave-53nh"
server_key: "6650c3d27fe515f741b3fb3726ee1f89"
server_key_iv: "4a90c43e9765c8a8683235092f52c969"
server_key_version: "encryptionkey@0"
I20260929 17:52:57.486961 506 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704375975685-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704375975685-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704375975685-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:52:57.499189 506 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:52:57.502306 506 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:52:57.502480 506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:52:57.503026 506 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 17:52:57.503826 506 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:52:57.503881 506 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:52:57.503935 506 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:52:57.503961 506 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 17:52:57.512458 506 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43697
I20260929 17:52:57.512475 634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43697 every 8 connection(s)
I20260929 17:52:57.513374 506 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704375975685-392-0/minicluster-data/ts-0/data/info.pb
I20260929 17:52:57.520139 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskup3Soe/build/debug/bin/kudu as pid 506
I20260929 17:52:57.520247 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704375975685-392-0/minicluster-data/ts-0/wal/instance
I20260929 17:52:57.520514 392 external_mini_cluster.cc:1428] Setting key 4c7ae9f855cf3fdd6b99d11d0cc435a3
Sep 29 17:52:57 dist-test-slave-53nh krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704377, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 17:52:57.527232 482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:52:57.515097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37147 (local address 127.0.98.62:38289)
0929 17:52:57.515235 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0929 17:52:57.515238 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 17:52:57.516098 (+ 860us) server_negotiation.cc:407] Connection header received
0929 17:52:57.517434 (+ 1336us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:52:57.517437 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:52:57.517493 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:52:57.517563 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:52:57.518724 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:52:57.519246 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:52:57.520531 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:52:57.520811 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:52:57.520905 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:52:57.523757 (+ 2852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:52:57.523760 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:52:57.523762 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:52:57.523787 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:52:57.525311 (+ 1524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:52:57.525755 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:52:57.525759 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:52:57.525760 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:52:57.525840 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:52:57.526115 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:52:57.526117 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:52:57.526119 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:52:57.526338 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:52:57.526437 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0929 17:52:57.526929 (+ 492us) server_negotiation.cc:299] Negotiation successful
0929 17:52:57.527058 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 17:52:57.527978 637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:52:57.515521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38289 (local address 127.0.98.1:37147)
0929 17:52:57.515953 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0929 17:52:57.515995 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 17:52:57.517222 (+ 1227us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:52:57.517738 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:52:57.517745 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:52:57.518107 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:52:57.518585 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:52:57.518597 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:52:57.519368 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:52:57.519370 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:52:57.520418 (+ 1048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:52:57.520429 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:52:57.520956 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:52:57.520959 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:52:57.521156 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:52:57.521897 (+ 741us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:52:57.521920 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:52:57.523590 (+ 1670us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:52:57.525410 (+ 1820us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:52:57.525415 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:52:57.525430 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:52:57.525668 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:52:57.525916 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:52:57.525918 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:52:57.525919 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:52:57.526041 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:52:57.526441 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:52:57.526447 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:52:57.526688 (+ 241us) client_negotiation.cc:769] Sending connection context
0929 17:52:57.526854 (+ 166us) client_negotiation.cc:240] Negotiation successful
0929 17:52:57.527111 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":112,"threads_started":1}
I20260929 17:52:57.529134 635 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38289
I20260929 17:52:57.529387 635 heartbeater.cc:464] Registering TS with master...
I20260929 17:52:57.529971 635 heartbeater.cc:511] Master 127.0.98.62:38289 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260929 17:52:57.531493 444 ts_manager.cc:194] Registered new tserver with Master: 1289dd7ce2bb4d99b099c5707cdf2fdc (127.0.98.1:43697)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260929 17:52:57.532712 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:37147
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704375975685-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704375975685-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260929 17:52:57.547773 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskup3Soe/build/debug/bin/kudu
/tmp/dist-test-taskup3Soe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704375975685-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704375975685-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704375975685-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704375975685-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38289
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704375975685-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704375975685-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704375975685-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704375975685-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:52:57.656957 640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:52:57.657269 640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:52:57.657333 640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:52:57.661546 640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:52:57.661623 640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:52:57.661722 640 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 17:52:57.667136 640 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704375975685-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704375975685-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704375975685-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704375975685-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38289
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.640
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704375975685-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:52:57.668190 640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:52:57.669220 640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:52:57.678892 645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:52:57.678887 646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:52:57.678903 648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:52:57.679045 640 server_base.cc:1034] running on GCE node
I20260929 17:52:57.679555 640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:52:57.680140 640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:52:57.681288 640 hybrid_clock.cc:648] HybridClock initialized: now 1790704377681275 us; error 37 us; skew 500 ppm
Sep 29 17:52:57 dist-test-slave-53nh krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704377, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:52:57.683817 640 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260929 17:52:57.684624 640 webserver.cc:460] Webserver started at http://127.0.98.2:46157/ using document root <none> and password file <none>
I20260929 17:52:57.685160 640 fs_manager.cc:359] Metadata directory not provided
I20260929 17:52:57.685220 640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:52:57.685384 640 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:52:57.687258 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704375975685-392-0/minicluster-data/ts-1/data/instance:
uuid: "7c2bcd131dcd4ebdb156fde1dc0ad2fd"
format_stamp: "Formatted at 2026-09-29 17:52:57 on dist-test-slave-53nh"
server_key: "1b19ba44777aacd75d9384936bd1daa2"
server_key_iv: "96a0deaf62f26a02501b6a91c8f89811"
server_key_version: "encryptionkey@0"
I20260929 17:52:57.687757 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704375975685-392-0/minicluster-data/ts-1/wal/instance:
uuid: "7c2bcd131dcd4ebdb156fde1dc0ad2fd"
format_stamp: "Formatted at 2026-09-29 17:52:57 on dist-test-slave-53nh"
server_key: "1b19ba44777aacd75d9384936bd1daa2"
server_key_iv: "96a0deaf62f26a02501b6a91c8f89811"
server_key_version: "encryptionkey@0"
I20260929 17:52:57.691525 640 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260929 17:52:57.693912 655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:52:57.695112 640 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 17:52:57.695258 640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704375975685-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704375975685-392-0/minicluster-data/ts-1/wal
uuid: "7c2bcd131dcd4ebdb156fde1dc0ad2fd"
format_stamp: "Formatted at 2026-09-29 17:52:57 on dist-test-slave-53nh"
server_key: "1b19ba44777aacd75d9384936bd1daa2"
server_key_iv: "96a0deaf62f26a02501b6a91c8f89811"
server_key_version: "encryptionkey@0"
I20260929 17:52:57.695351 640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704375975685-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704375975685-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704375975685-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:52:57.712602 640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:52:57.715243 640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:52:57.715413 640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:52:57.715941 640 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 17:52:57.716710 640 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:52:57.716755 640 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:52:57.716799 640 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:52:57.716826 640 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 17:52:57.725781 640 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35601
I20260929 17:52:57.725811 768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35601 every 8 connection(s)
I20260929 17:52:57.726646 640 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704375975685-392-0/minicluster-data/ts-1/data/info.pb
I20260929 17:52:57.733279 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskup3Soe/build/debug/bin/kudu as pid 640
I20260929 17:52:57.733387 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704375975685-392-0/minicluster-data/ts-1/wal/instance
I20260929 17:52:57.733611 392 external_mini_cluster.cc:1428] Setting key 3133906e5d5086fd77b9aeb941fbf088
Sep 29 17:52:57 dist-test-slave-53nh krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704377, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 17:52:57.739219 482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:52:57.728286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34075 (local address 127.0.98.62:38289)
0929 17:52:57.728440 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0929 17:52:57.728445 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:52:57.729261 (+ 816us) server_negotiation.cc:407] Connection header received
0929 17:52:57.730525 (+ 1264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:52:57.730528 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:52:57.730575 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:52:57.730645 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:52:57.731708 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:52:57.732261 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:52:57.733485 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:52:57.733708 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:52:57.733810 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:52:57.736116 (+ 2306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:52:57.736119 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:52:57.736121 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:52:57.736142 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:52:57.737423 (+ 1281us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:52:57.737864 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:52:57.737867 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:52:57.737869 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:52:57.737942 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:52:57.738230 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:52:57.738233 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:52:57.738235 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:52:57.738402 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:52:57.738484 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0929 17:52:57.738946 (+ 462us) server_negotiation.cc:299] Negotiation successful
0929 17:52:57.739059 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 17:52:57.740084 771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:52:57.728707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38289 (local address 127.0.98.2:34075)
0929 17:52:57.729134 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0929 17:52:57.729174 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 17:52:57.730336 (+ 1162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:52:57.730800 (+ 464us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:52:57.730806 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:52:57.731143 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:52:57.731600 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:52:57.731612 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:52:57.732350 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:52:57.732352 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:52:57.733329 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:52:57.733339 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:52:57.733842 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:52:57.733846 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:52:57.734027 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:52:57.734584 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:52:57.734603 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:52:57.735967 (+ 1364us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:52:57.737524 (+ 1557us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:52:57.737530 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:52:57.737545 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:52:57.737776 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:52:57.738019 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:52:57.738022 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:52:57.738024 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:52:57.738150 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:52:57.738492 (+ 342us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:52:57.738498 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:52:57.738729 (+ 231us) client_negotiation.cc:769] Sending connection context
0929 17:52:57.738887 (+ 158us) client_negotiation.cc:240] Negotiation successful
0929 17:52:57.739193 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":103,"threads_started":1}
I20260929 17:52:57.741530 769 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38289
I20260929 17:52:57.741751 769 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260929 17:52:57.742234 769 heartbeater.cc:511] Master 127.0.98.62:38289 requested a full tablet report, sending...
I20260929 17:52:57.743294 444 ts_manager.cc:194] Registered new tserver with Master: 7c2bcd131dcd4ebdb156fde1dc0ad2fd (127.0.98.2:35601)
I20260929 17:52:57.743825 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:34075
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704375975685-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704375975685-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260929 17:52:57.758154 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskup3Soe/build/debug/bin/kudu
/tmp/dist-test-taskup3Soe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704375975685-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704375975685-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704375975685-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704375975685-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38289
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704375975685-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704375975685-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704375975685-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704375975685-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:52:57.866046 774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:52:57.866313 774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:52:57.866358 774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:52:57.870631 774 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:52:57.870688 774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:52:57.870769 774 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 17:52:57.876017 774 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704375975685-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704375975685-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704375975685-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704375975685-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38289
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.774
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704375975685-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:52:57.876989 774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:52:57.878002 774 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:52:57.887008 779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:52:57.887008 782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:52:57.887058 780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:52:57.887331 774 server_base.cc:1034] running on GCE node
I20260929 17:52:57.887738 774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:52:57.888352 774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:52:57.889523 774 hybrid_clock.cc:648] HybridClock initialized: now 1790704377889501 us; error 32 us; skew 500 ppm
Sep 29 17:52:57 dist-test-slave-53nh krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704377, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:52:57.892084 774 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260929 17:52:57.892949 774 webserver.cc:460] Webserver started at http://127.0.98.3:39389/ using document root <none> and password file <none>
I20260929 17:52:57.893486 774 fs_manager.cc:359] Metadata directory not provided
I20260929 17:52:57.893548 774 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:52:57.893728 774 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:52:57.895686 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704375975685-392-0/minicluster-data/ts-2/data/instance:
uuid: "66988486a40d4b36a09c0835365a4797"
format_stamp: "Formatted at 2026-09-29 17:52:57 on dist-test-slave-53nh"
server_key: "88129311e422d4670ff748faa42afb9a"
server_key_iv: "1c8fa62a34baca01f8d2cd0c3a6af334"
server_key_version: "encryptionkey@0"
I20260929 17:52:57.896215 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704375975685-392-0/minicluster-data/ts-2/wal/instance:
uuid: "66988486a40d4b36a09c0835365a4797"
format_stamp: "Formatted at 2026-09-29 17:52:57 on dist-test-slave-53nh"
server_key: "88129311e422d4670ff748faa42afb9a"
server_key_iv: "1c8fa62a34baca01f8d2cd0c3a6af334"
server_key_version: "encryptionkey@0"
I20260929 17:52:57.899755 774 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260929 17:52:57.902076 789 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:52:57.903263 774 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 17:52:57.903365 774 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704375975685-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704375975685-392-0/minicluster-data/ts-2/wal
uuid: "66988486a40d4b36a09c0835365a4797"
format_stamp: "Formatted at 2026-09-29 17:52:57 on dist-test-slave-53nh"
server_key: "88129311e422d4670ff748faa42afb9a"
server_key_iv: "1c8fa62a34baca01f8d2cd0c3a6af334"
server_key_version: "encryptionkey@0"
I20260929 17:52:57.903455 774 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704375975685-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704375975685-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704375975685-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:52:57.917274 774 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:52:57.920017 774 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:52:57.920176 774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:52:57.920738 774 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 17:52:57.921551 774 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:52:57.921604 774 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:52:57.921656 774 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:52:57.921680 774 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 17:52:57.930728 774 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33635
I20260929 17:52:57.930747 902 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33635 every 8 connection(s)
I20260929 17:52:57.931574 774 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704375975685-392-0/minicluster-data/ts-2/data/info.pb
I20260929 17:52:57.933559 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskup3Soe/build/debug/bin/kudu as pid 774
I20260929 17:52:57.933648 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790704375975685-392-0/minicluster-data/ts-2/wal/instance
I20260929 17:52:57.933907 392 external_mini_cluster.cc:1428] Setting key a238b93bce08fe4d25dd62d08e00d1b0
Sep 29 17:52:57 dist-test-slave-53nh krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704377, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 17:52:57.944084 482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:52:57.933241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41175 (local address 127.0.98.62:38289)
0929 17:52:57.933389 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0929 17:52:57.933393 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:52:57.934269 (+ 876us) server_negotiation.cc:407] Connection header received
0929 17:52:57.935792 (+ 1523us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:52:57.935795 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:52:57.935845 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:52:57.935932 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:52:57.937152 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:52:57.937670 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:52:57.938735 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:52:57.938941 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:52:57.939008 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:52:57.941220 (+ 2212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:52:57.941223 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:52:57.941225 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:52:57.941250 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:52:57.942453 (+ 1203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:52:57.942871 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:52:57.942873 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:52:57.942874 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:52:57.942947 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:52:57.943191 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:52:57.943193 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:52:57.943194 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:52:57.943363 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:52:57.943438 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0929 17:52:57.943858 (+ 420us) server_negotiation.cc:299] Negotiation successful
0929 17:52:57.943965 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 17:52:57.944864 905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:52:57.933607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38289 (local address 127.0.98.3:41175)
0929 17:52:57.934098 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0929 17:52:57.934153 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0929 17:52:57.935606 (+ 1453us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:52:57.936098 (+ 492us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:52:57.936104 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:52:57.936524 (+ 420us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:52:57.936991 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:52:57.937037 (+ 46us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:52:57.937761 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:52:57.937763 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:52:57.938635 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:52:57.938643 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:52:57.939041 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:52:57.939043 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:52:57.939174 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:52:57.939672 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:52:57.939687 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:52:57.941069 (+ 1382us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:52:57.942555 (+ 1486us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:52:57.942560 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:52:57.942572 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:52:57.942800 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:52:57.943009 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:52:57.943011 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:52:57.943013 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:52:57.943123 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:52:57.943433 (+ 310us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:52:57.943437 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:52:57.943659 (+ 222us) client_negotiation.cc:769] Sending connection context
0929 17:52:57.943806 (+ 147us) client_negotiation.cc:240] Negotiation successful
0929 17:52:57.944055 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":112,"threads_started":1}
I20260929 17:52:57.945984 903 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38289
I20260929 17:52:57.946205 903 heartbeater.cc:464] Registering TS with master...
I20260929 17:52:57.946657 903 heartbeater.cc:511] Master 127.0.98.62:38289 requested a full tablet report, sending...
I20260929 17:52:57.947662 444 ts_manager.cc:194] Registered new tserver with Master: 66988486a40d4b36a09c0835365a4797 (127.0.98.3:33635)
I20260929 17:52:57.948225 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:41175
I20260929 17:52:57.959734 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 17:52:57 dist-test-slave-53nh krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704377, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 17:52:57 dist-test-slave-53nh krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704377, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260929 17:52:57.984016 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:52:57.975189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53750 (local address 127.0.98.1:43697)
0929 17:52:57.975493 (+ 304us) server_negotiation.cc:206] Beginning negotiation
0929 17:52:57.975504 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 17:52:57.975529 (+ 25us) server_negotiation.cc:407] Connection header received
0929 17:52:57.975630 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:52:57.975638 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:52:57.975846 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:52:57.975976 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:52:57.976608 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:52:57.977462 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:52:57.978285 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:52:57.978639 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:52:57.978741 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:52:57.980883 (+ 2142us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:52:57.980891 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:52:57.980903 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:52:57.980926 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:52:57.982390 (+ 1464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:52:57.982763 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:52:57.982768 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:52:57.982769 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:52:57.982853 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:52:57.983085 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:52:57.983088 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:52:57.983089 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:52:57.983345 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:52:57.983520 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0929 17:52:57.983736 (+ 216us) server_negotiation.cc:299] Negotiation successful
0929 17:52:57.983856 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":200,"thread_start_us":94,"threads_started":1}
W20260929 17:52:57.984994 549 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53750
Sep 29 17:52:57 dist-test-slave-53nh krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704377, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 17:52:58 dist-test-slave-53nh krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704377, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260929 17:52:58.008903 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:52:58.000992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53760 (local address 127.0.98.1:43697)
0929 17:52:58.001140 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0929 17:52:58.001145 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:52:58.001263 (+ 118us) server_negotiation.cc:407] Connection header received
0929 17:52:58.001386 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:52:58.001389 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:52:58.001430 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:52:58.001498 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:52:58.002132 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:52:58.002672 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:52:58.003430 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:52:58.003646 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:52:58.003718 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:52:58.005968 (+ 2250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:52:58.005972 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:52:58.005975 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:52:58.005999 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:52:58.007407 (+ 1408us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:52:58.007763 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:52:58.007765 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:52:58.007767 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:52:58.007832 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:52:58.008079 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:52:58.008083 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:52:58.008084 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:52:58.008309 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:52:58.008421 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0929 17:52:58.008634 (+ 213us) server_negotiation.cc:299] Negotiation successful
0929 17:52:58.008745 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260929 17:52:58.009867 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskup3Soe/build/debug/bin/kudu with pid 506
I20260929 17:52:58.015056 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskup3Soe/build/debug/bin/kudu with pid 640
I20260929 17:52:58.021483 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskup3Soe/build/debug/bin/kudu with pid 774
I20260929 17:52:58.026366 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskup3Soe/build/debug/bin/kudu with pid 412
2026-09-29T17:52:58Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2042 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704375975685-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 17:52:58 dist-test-slave-53nh krb5kdc[917](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 17:52:58 dist-test-slave-53nh krb5kdc[917](info): set up 2 sockets
Sep 29 17:52:58 dist-test-slave-53nh krb5kdc[917](info): commencing operation
krb5kdc: starting...
W20260929 17:52:58.976488 392 mini_kdc.cc:121] Time spent starting KDC: real 0.930s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704375975685-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704375975685-392-0/krb5kdc/test-user.keytab.
Sep 29 17:52:59 dist-test-slave-53nh krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704379, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T17:52:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T17:52:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704375975685-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704375975685-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 17:52:59.091169 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskup3Soe/build/debug/bin/kudu
/tmp/dist-test-taskup3Soe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704375975685-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704375975685-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704375975685-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704375975685-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37249
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704375975685-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37249
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704375975685-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704375975685-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704375975685-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:52:59.198190 930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:52:59.198451 930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:52:59.198504 930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:52:59.202447 930 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 17:52:59.202507 930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:52:59.202531 930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:52:59.202553 930 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 17:52:59.202580 930 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 17:52:59.207823 930 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704375975685-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704375975685-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37249
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704375975685-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37249
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704375975685-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.930
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704375975685-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:52:59.208840 930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:52:59.209849 930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:52:59.217934 936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:52:59.217952 935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:52:59.218039 930 server_base.cc:1034] running on GCE node
W20260929 17:52:59.217953 938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:52:59.218730 930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:52:59.219753 930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:52:59.220924 930 hybrid_clock.cc:648] HybridClock initialized: now 1790704379220887 us; error 52 us; skew 500 ppm
Sep 29 17:52:59 dist-test-slave-53nh krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704379, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:52:59.223635 930 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 17:52:59.224511 930 webserver.cc:460] Webserver started at http://127.0.98.62:43787/ using document root <none> and password file <none>
I20260929 17:52:59.225064 930 fs_manager.cc:359] Metadata directory not provided
I20260929 17:52:59.225139 930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:52:59.225364 930 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:52:59.227324 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704375975685-392-0/minicluster-data/master-0/data/instance:
uuid: "c9237015b2e344cead88301759f6eccc"
format_stamp: "Formatted at 2026-09-29 17:52:59 on dist-test-slave-53nh"
server_key: "46f064fbcdbd71ba870573022883e267"
server_key_iv: "a9aad0e927ea3e0375a10800f23c293a"
server_key_version: "encryptionkey@0"
I20260929 17:52:59.227838 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704375975685-392-0/minicluster-data/master-0/wal/instance:
uuid: "c9237015b2e344cead88301759f6eccc"
format_stamp: "Formatted at 2026-09-29 17:52:59 on dist-test-slave-53nh"
server_key: "46f064fbcdbd71ba870573022883e267"
server_key_iv: "a9aad0e927ea3e0375a10800f23c293a"
server_key_version: "encryptionkey@0"
I20260929 17:52:59.231554 930 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 17:52:59.233974 945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:52:59.235200 930 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 17:52:59.235323 930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704375975685-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704375975685-392-0/minicluster-data/master-0/wal
uuid: "c9237015b2e344cead88301759f6eccc"
format_stamp: "Formatted at 2026-09-29 17:52:59 on dist-test-slave-53nh"
server_key: "46f064fbcdbd71ba870573022883e267"
server_key_iv: "a9aad0e927ea3e0375a10800f23c293a"
server_key_version: "encryptionkey@0"
I20260929 17:52:59.235428 930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704375975685-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704375975685-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704375975685-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:52:59.258011 930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:52:59.260973 930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:52:59.261184 930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:52:59.267685 930 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37249
I20260929 17:52:59.267747 997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37249 every 8 connection(s)
I20260929 17:52:59.268637 930 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704375975685-392-0/minicluster-data/master-0/data/info.pb
I20260929 17:52:59.271404 998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:52:59.276767 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c9237015b2e344cead88301759f6eccc: Bootstrap starting.
I20260929 17:52:59.277045 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskup3Soe/build/debug/bin/kudu as pid 930
I20260929 17:52:59.277143 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704375975685-392-0/minicluster-data/master-0/wal/instance
I20260929 17:52:59.277382 392 external_mini_cluster.cc:1428] Setting key 6cda4ed1e7975b90ad2f592802a9c84d
I20260929 17:52:59.279325 998 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c9237015b2e344cead88301759f6eccc: Neither blocks nor log segments found. Creating new log.
I20260929 17:52:59.280179 998 log.cc:824] T 00000000000000000000000000000000 P c9237015b2e344cead88301759f6eccc: Log is configured to *not* fsync() on all Append() calls
I20260929 17:52:59.282291 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c9237015b2e344cead88301759f6eccc: No bootstrap required, opened a new log
I20260929 17:52:59.286069 998 raft_consensus.cc:348] T 00000000000000000000000000000000 P c9237015b2e344cead88301759f6eccc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9237015b2e344cead88301759f6eccc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37249 } }
Sep 29 17:52:59 dist-test-slave-53nh krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704379, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 17:52:59.286286 998 raft_consensus.cc:374] T 00000000000000000000000000000000 P c9237015b2e344cead88301759f6eccc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:52:59.286320 998 raft_consensus.cc:729] T 00000000000000000000000000000000 P c9237015b2e344cead88301759f6eccc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9237015b2e344cead88301759f6eccc, State: Initialized, Role: FOLLOWER
I20260929 17:52:59.286813 998 consensus_queue.cc:260] T 00000000000000000000000000000000 P c9237015b2e344cead88301759f6eccc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9237015b2e344cead88301759f6eccc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37249 } }
I20260929 17:52:59.286964 998 raft_consensus.cc:388] T 00000000000000000000000000000000 P c9237015b2e344cead88301759f6eccc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 17:52:59.287029 998 raft_consensus.cc:482] T 00000000000000000000000000000000 P c9237015b2e344cead88301759f6eccc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 17:52:59.287118 998 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c9237015b2e344cead88301759f6eccc [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:52:59.287937 998 raft_consensus.cc:504] T 00000000000000000000000000000000 P c9237015b2e344cead88301759f6eccc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9237015b2e344cead88301759f6eccc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37249 } }
I20260929 17:52:59.288255 998 leader_election.cc:302] T 00000000000000000000000000000000 P c9237015b2e344cead88301759f6eccc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c9237015b2e344cead88301759f6eccc; no voters:
I20260929 17:52:59.288524 998 leader_election.cc:288] T 00000000000000000000000000000000 P c9237015b2e344cead88301759f6eccc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 17:52:59.288630 1003 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c9237015b2e344cead88301759f6eccc [term 1 FOLLOWER]: Leader election won for term 1
I20260929 17:52:59.288854 1003 raft_consensus.cc:686] T 00000000000000000000000000000000 P c9237015b2e344cead88301759f6eccc [term 1 LEADER]: Becoming Leader. State: Replica: c9237015b2e344cead88301759f6eccc, State: Running, Role: LEADER
I20260929 17:52:59.289206 1003 consensus_queue.cc:237] T 00000000000000000000000000000000 P c9237015b2e344cead88301759f6eccc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9237015b2e344cead88301759f6eccc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37249 } }
I20260929 17:52:59.289798 998 sys_catalog.cc:564] T 00000000000000000000000000000000 P c9237015b2e344cead88301759f6eccc [sys.catalog]: configured and running, proceeding with master startup.
I20260929 17:52:59.290755 1005 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9237015b2e344cead88301759f6eccc [sys.catalog]: SysCatalogTable state changed. Reason: New leader c9237015b2e344cead88301759f6eccc. Latest consensus state: current_term: 1 leader_uuid: "c9237015b2e344cead88301759f6eccc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9237015b2e344cead88301759f6eccc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37249 } } }
I20260929 17:52:59.290890 1005 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9237015b2e344cead88301759f6eccc [sys.catalog]: This master's current role is: LEADER
I20260929 17:52:59.291213 1004 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9237015b2e344cead88301759f6eccc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c9237015b2e344cead88301759f6eccc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9237015b2e344cead88301759f6eccc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37249 } } }
I20260929 17:52:59.291318 1004 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9237015b2e344cead88301759f6eccc [sys.catalog]: This master's current role is: LEADER
I20260929 17:52:59.292582 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:52:59.278697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50186 (local address 127.0.98.62:37249)
0929 17:52:59.279121 (+ 424us) server_negotiation.cc:206] Beginning negotiation
0929 17:52:59.279139 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 17:52:59.279180 (+ 41us) server_negotiation.cc:407] Connection header received
0929 17:52:59.280008 (+ 828us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:52:59.280031 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:52:59.280449 (+ 418us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:52:59.280780 (+ 331us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:52:59.281497 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:52:59.282894 (+ 1397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:52:59.283689 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:52:59.284390 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:52:59.284543 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:52:59.286928 (+ 2385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:52:59.286934 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:52:59.286953 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:52:59.286985 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:52:59.288682 (+ 1697us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:52:59.289217 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:52:59.289224 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:52:59.289230 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:52:59.289375 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:52:59.289678 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:52:59.289681 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:52:59.289683 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:52:59.290156 (+ 473us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:52:59.290359 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0929 17:52:59.291124 (+ 765us) server_negotiation.cc:299] Negotiation successful
0929 17:52:59.291449 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":105,"threads_started":1}
W20260929 17:52:59.294780 1018 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c9237015b2e344cead88301759f6eccc: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 17:52:59.294855 1018 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 17:52:59.294955 1012 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 17:52:59.295615 1012 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 17:52:59.300464 1012 catalog_manager.cc:1347] Generated new cluster ID: c9ad31a416bb43968b298bc7a90f60fb
I20260929 17:52:59.300527 1012 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 17:52:59.313683 1012 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 17:52:59.315622 1012 catalog_manager.cc:1504] Loading token signing keys...
I20260929 17:52:59.320226 1012 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c9237015b2e344cead88301759f6eccc: Generated new TSK 0
I20260929 17:52:59.321039 1012 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704375975685-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704375975685-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260929 17:52:59.348713 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskup3Soe/build/debug/bin/kudu
/tmp/dist-test-taskup3Soe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704375975685-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704375975685-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704375975685-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704375975685-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37249
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704375975685-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704375975685-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704375975685-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704375975685-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:52:59.459484 1024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:52:59.459758 1024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:52:59.459805 1024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:52:59.463837 1024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:52:59.463915 1024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:52:59.463990 1024 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 17:52:59.469296 1024 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704375975685-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704375975685-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704375975685-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704375975685-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37249
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1024
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704375975685-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:52:59.470389 1024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:52:59.471436 1024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:52:59.480844 1029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:52:59.480923 1024 server_base.cc:1034] running on GCE node
W20260929 17:52:59.480899 1032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:52:59.481122 1030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:52:59.481534 1024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:52:59.482129 1024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:52:59.483292 1024 hybrid_clock.cc:648] HybridClock initialized: now 1790704379483268 us; error 35 us; skew 500 ppm
Sep 29 17:52:59 dist-test-slave-53nh krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704379, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:52:59.485903 1024 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260929 17:52:59.486701 1024 webserver.cc:460] Webserver started at http://127.0.98.1:34759/ using document root <none> and password file <none>
I20260929 17:52:59.487229 1024 fs_manager.cc:359] Metadata directory not provided
I20260929 17:52:59.487290 1024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:52:59.487447 1024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:52:59.489434 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704375975685-392-0/minicluster-data/ts-0/data/instance:
uuid: "23d4f46ee6a9414bb0b0408f1a4da527"
format_stamp: "Formatted at 2026-09-29 17:52:59 on dist-test-slave-53nh"
server_key: "f3c44af953b9dfc1dac77b0defa3cbdc"
server_key_iv: "bbcd30f690f7f13f34c758232bf0ef61"
server_key_version: "encryptionkey@0"
I20260929 17:52:59.489986 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704375975685-392-0/minicluster-data/ts-0/wal/instance:
uuid: "23d4f46ee6a9414bb0b0408f1a4da527"
format_stamp: "Formatted at 2026-09-29 17:52:59 on dist-test-slave-53nh"
server_key: "f3c44af953b9dfc1dac77b0defa3cbdc"
server_key_iv: "bbcd30f690f7f13f34c758232bf0ef61"
server_key_version: "encryptionkey@0"
I20260929 17:52:59.493643 1024 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 17:52:59.495983 1039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:52:59.497253 1024 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 17:52:59.497349 1024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704375975685-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704375975685-392-0/minicluster-data/ts-0/wal
uuid: "23d4f46ee6a9414bb0b0408f1a4da527"
format_stamp: "Formatted at 2026-09-29 17:52:59 on dist-test-slave-53nh"
server_key: "f3c44af953b9dfc1dac77b0defa3cbdc"
server_key_iv: "bbcd30f690f7f13f34c758232bf0ef61"
server_key_version: "encryptionkey@0"
I20260929 17:52:59.497444 1024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704375975685-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704375975685-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704375975685-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:52:59.518541 1024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:52:59.521714 1024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:52:59.521896 1024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:52:59.522464 1024 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 17:52:59.523250 1024 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:52:59.523295 1024 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:52:59.523339 1024 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:52:59.523366 1024 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 17:52:59.531841 1024 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40803
I20260929 17:52:59.531862 1152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40803 every 8 connection(s)
I20260929 17:52:59.532691 1024 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704375975685-392-0/minicluster-data/ts-0/data/info.pb
I20260929 17:52:59.534708 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskup3Soe/build/debug/bin/kudu as pid 1024
I20260929 17:52:59.534802 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704375975685-392-0/minicluster-data/ts-0/wal/instance
I20260929 17:52:59.535035 392 external_mini_cluster.cc:1428] Setting key d9ee60d37993f5ebf0ed5127c589e1f6
Sep 29 17:52:59 dist-test-slave-53nh krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704379, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260929 17:52:59.547648 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:52:59.534398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35839 (local address 127.0.98.62:37249)
0929 17:52:59.534531 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0929 17:52:59.534535 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:52:59.535473 (+ 938us) server_negotiation.cc:407] Connection header received
0929 17:52:59.537084 (+ 1611us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:52:59.537087 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:52:59.537129 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:52:59.537188 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:52:59.538622 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:52:59.539257 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:52:59.540892 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:52:59.541168 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:52:59.541249 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:52:59.544129 (+ 2880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:52:59.544133 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:52:59.544135 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:52:59.544160 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:52:59.545522 (+ 1362us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:52:59.546057 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:52:59.546060 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:52:59.546061 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:52:59.546143 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:52:59.546528 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:52:59.546531 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:52:59.546533 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:52:59.546759 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:52:59.546839 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0929 17:52:59.547358 (+ 519us) server_negotiation.cc:299] Negotiation successful
0929 17:52:59.547481 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 17:52:59.548457 1155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:52:59.534810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37249 (local address 127.0.98.1:35839)
0929 17:52:59.535303 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0929 17:52:59.535361 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0929 17:52:59.536824 (+ 1463us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:52:59.537393 (+ 569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:52:59.537403 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:52:59.537884 (+ 481us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:52:59.538479 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:52:59.538498 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:52:59.539376 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:52:59.539379 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:52:59.540771 (+ 1392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:52:59.540782 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:52:59.541315 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:52:59.541319 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:52:59.541526 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:52:59.542123 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:52:59.542143 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:52:59.543955 (+ 1812us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:52:59.545641 (+ 1686us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:52:59.545646 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:52:59.545657 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:52:59.545945 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:52:59.546268 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:52:59.546271 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:52:59.546272 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:52:59.546419 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:52:59.546866 (+ 447us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:52:59.546873 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:52:59.547114 (+ 241us) client_negotiation.cc:769] Sending connection context
0929 17:52:59.547300 (+ 186us) client_negotiation.cc:240] Negotiation successful
0929 17:52:59.547658 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":104,"threads_started":1}
I20260929 17:52:59.549750 1153 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37249
I20260929 17:52:59.549984 1153 heartbeater.cc:464] Registering TS with master...
I20260929 17:52:59.550522 1153 heartbeater.cc:511] Master 127.0.98.62:37249 requested a full tablet report, sending...
I20260929 17:52:59.552237 962 ts_manager.cc:194] Registered new tserver with Master: 23d4f46ee6a9414bb0b0408f1a4da527 (127.0.98.1:40803)
I20260929 17:52:59.553840 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:35839
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704375975685-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704375975685-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260929 17:52:59.562012 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskup3Soe/build/debug/bin/kudu
/tmp/dist-test-taskup3Soe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704375975685-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704375975685-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704375975685-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704375975685-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37249
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704375975685-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704375975685-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704375975685-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704375975685-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:52:59.673166 1158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:52:59.673445 1158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:52:59.673494 1158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:52:59.677908 1158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:52:59.677975 1158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:52:59.678051 1158 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 17:52:59.683426 1158 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704375975685-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704375975685-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704375975685-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704375975685-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37249
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1158
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704375975685-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:52:59.684491 1158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:52:59.685539 1158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:52:59.695119 1164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:52:59.695119 1166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:52:59.695122 1163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:52:59.695747 1158 server_base.cc:1034] running on GCE node
I20260929 17:52:59.696090 1158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:52:59.696693 1158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:52:59.697858 1158 hybrid_clock.cc:648] HybridClock initialized: now 1790704379697838 us; error 32 us; skew 500 ppm
Sep 29 17:52:59 dist-test-slave-53nh krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704379, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:52:59.700488 1158 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260929 17:52:59.701412 1158 webserver.cc:460] Webserver started at http://127.0.98.2:39923/ using document root <none> and password file <none>
I20260929 17:52:59.701936 1158 fs_manager.cc:359] Metadata directory not provided
I20260929 17:52:59.701998 1158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:52:59.702174 1158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:52:59.704190 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704375975685-392-0/minicluster-data/ts-1/data/instance:
uuid: "0f026596c3354a0ca27050ad958baed6"
format_stamp: "Formatted at 2026-09-29 17:52:59 on dist-test-slave-53nh"
server_key: "1e5cb9b36fe6b61061721216521180ac"
server_key_iv: "6c2e06b04dd97a75be3bf1666a93e34b"
server_key_version: "encryptionkey@0"
I20260929 17:52:59.704717 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704375975685-392-0/minicluster-data/ts-1/wal/instance:
uuid: "0f026596c3354a0ca27050ad958baed6"
format_stamp: "Formatted at 2026-09-29 17:52:59 on dist-test-slave-53nh"
server_key: "1e5cb9b36fe6b61061721216521180ac"
server_key_iv: "6c2e06b04dd97a75be3bf1666a93e34b"
server_key_version: "encryptionkey@0"
I20260929 17:52:59.708413 1158 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260929 17:52:59.710824 1173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:52:59.712409 1158 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260929 17:52:59.712513 1158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704375975685-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704375975685-392-0/minicluster-data/ts-1/wal
uuid: "0f026596c3354a0ca27050ad958baed6"
format_stamp: "Formatted at 2026-09-29 17:52:59 on dist-test-slave-53nh"
server_key: "1e5cb9b36fe6b61061721216521180ac"
server_key_iv: "6c2e06b04dd97a75be3bf1666a93e34b"
server_key_version: "encryptionkey@0"
I20260929 17:52:59.712606 1158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704375975685-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704375975685-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704375975685-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:52:59.735942 1158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:52:59.739336 1158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:52:59.739514 1158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:52:59.740116 1158 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 17:52:59.740943 1158 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:52:59.740991 1158 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:52:59.741086 1158 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:52:59.741110 1158 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 17:52:59.750329 1158 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42023
I20260929 17:52:59.750349 1286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42023 every 8 connection(s)
I20260929 17:52:59.751226 1158 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704375975685-392-0/minicluster-data/ts-1/data/info.pb
I20260929 17:52:59.757898 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskup3Soe/build/debug/bin/kudu as pid 1158
I20260929 17:52:59.758026 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704375975685-392-0/minicluster-data/ts-1/wal/instance
I20260929 17:52:59.758353 392 external_mini_cluster.cc:1428] Setting key 3476939945cc9c3a4b58383c783baa86
Sep 29 17:52:59 dist-test-slave-53nh krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704379, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 17:52:59.765867 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:52:59.753146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54637 (local address 127.0.98.62:37249)
0929 17:52:59.753274 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0929 17:52:59.753277 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 17:52:59.754252 (+ 975us) server_negotiation.cc:407] Connection header received
0929 17:52:59.755542 (+ 1290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:52:59.755545 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:52:59.755594 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:52:59.755679 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:52:59.756977 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:52:59.757583 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:52:59.759090 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:52:59.759321 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:52:59.759399 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:52:59.762281 (+ 2882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:52:59.762284 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:52:59.762286 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:52:59.762311 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:52:59.763773 (+ 1462us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:52:59.764287 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:52:59.764291 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:52:59.764293 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:52:59.764382 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:52:59.764686 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:52:59.764689 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:52:59.764691 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:52:59.764942 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:52:59.765092 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0929 17:52:59.765565 (+ 473us) server_negotiation.cc:299] Negotiation successful
0929 17:52:59.765696 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 17:52:59.766671 1289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:52:59.753553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37249 (local address 127.0.98.2:54637)
0929 17:52:59.754051 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0929 17:52:59.754094 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 17:52:59.755322 (+ 1228us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:52:59.755856 (+ 534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:52:59.755864 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:52:59.756274 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:52:59.756812 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:52:59.756828 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:52:59.757747 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:52:59.757752 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:52:59.758987 (+ 1235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:52:59.758997 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:52:59.759459 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:52:59.759464 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:52:59.759628 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:52:59.760218 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:52:59.760239 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:52:59.762077 (+ 1838us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:52:59.763923 (+ 1846us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:52:59.763928 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:52:59.763940 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:52:59.764190 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:52:59.764470 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:52:59.764472 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:52:59.764473 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:52:59.764609 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:52:59.765078 (+ 469us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:52:59.765083 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:52:59.765331 (+ 248us) client_negotiation.cc:769] Sending connection context
0929 17:52:59.765495 (+ 164us) client_negotiation.cc:240] Negotiation successful
0929 17:52:59.765779 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":130,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260929 17:52:59.767863 1287 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37249
I20260929 17:52:59.768126 1287 heartbeater.cc:464] Registering TS with master...
I20260929 17:52:59.768610 1287 heartbeater.cc:511] Master 127.0.98.62:37249 requested a full tablet report, sending...
I20260929 17:52:59.769739 962 ts_manager.cc:194] Registered new tserver with Master: 0f026596c3354a0ca27050ad958baed6 (127.0.98.2:42023)
I20260929 17:52:59.770313 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:54637
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260929 17:53:00.556281 1153 heartbeater.cc:503] Master 127.0.98.62:37249 was elected leader, sending a full tablet report...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704375975685-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704375975685-392-0/krb5kdc/kudu_127.0.98.3.keytab.
W20260929 17:53:00.569612 392 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.98.3: real 0.810s user 0.000s sys 0.003s
I20260929 17:53:00.570150 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskup3Soe/build/debug/bin/kudu
/tmp/dist-test-taskup3Soe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704375975685-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704375975685-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704375975685-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704375975685-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37249
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704375975685-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704375975685-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704375975685-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704375975685-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:53:00.680457 1292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:53:00.680716 1292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:53:00.680760 1292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:53:00.685041 1292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:53:00.685132 1292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:53:00.685200 1292 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 17:53:00.690796 1292 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704375975685-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704375975685-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704375975685-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704375975685-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37249
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1292
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704375975685-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:53:00.691879 1292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:53:00.692893 1292 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:53:00.702466 1298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:53:00.702533 1297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:53:00.702543 1292 server_base.cc:1034] running on GCE node
W20260929 17:53:00.702486 1300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:53:00.703125 1292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:53:00.703747 1292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:53:00.704926 1292 hybrid_clock.cc:648] HybridClock initialized: now 1790704380704906 us; error 34 us; skew 500 ppm
Sep 29 17:53:00 dist-test-slave-53nh krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704380, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:53:00.707751 1292 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260929 17:53:00.708603 1292 webserver.cc:460] Webserver started at http://127.0.98.3:34943/ using document root <none> and password file <none>
I20260929 17:53:00.709160 1292 fs_manager.cc:359] Metadata directory not provided
I20260929 17:53:00.709223 1292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:53:00.709393 1292 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:53:00.711347 1292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704375975685-392-0/minicluster-data/ts-2/data/instance:
uuid: "4cf0cb525aab4b19984f91feed8af780"
format_stamp: "Formatted at 2026-09-29 17:53:00 on dist-test-slave-53nh"
server_key: "e8408bc7a051c425329e8cbeecc622a8"
server_key_iv: "74883c24c28a76c56956a2a51fd8f213"
server_key_version: "encryptionkey@0"
I20260929 17:53:00.711839 1292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704375975685-392-0/minicluster-data/ts-2/wal/instance:
uuid: "4cf0cb525aab4b19984f91feed8af780"
format_stamp: "Formatted at 2026-09-29 17:53:00 on dist-test-slave-53nh"
server_key: "e8408bc7a051c425329e8cbeecc622a8"
server_key_iv: "74883c24c28a76c56956a2a51fd8f213"
server_key_version: "encryptionkey@0"
I20260929 17:53:00.715369 1292 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260929 17:53:00.717708 1307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:53:00.718852 1292 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 17:53:00.718961 1292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704375975685-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704375975685-392-0/minicluster-data/ts-2/wal
uuid: "4cf0cb525aab4b19984f91feed8af780"
format_stamp: "Formatted at 2026-09-29 17:53:00 on dist-test-slave-53nh"
server_key: "e8408bc7a051c425329e8cbeecc622a8"
server_key_iv: "74883c24c28a76c56956a2a51fd8f213"
server_key_version: "encryptionkey@0"
I20260929 17:53:00.719054 1292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704375975685-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704375975685-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704375975685-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:53:00.737830 1292 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:53:00.741106 1292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:53:00.741300 1292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:53:00.741894 1292 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 17:53:00.742714 1292 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:53:00.742761 1292 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:53:00.742820 1292 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:53:00.742844 1292 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 17:53:00.751497 1292 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:32967
I20260929 17:53:00.751521 1420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:32967 every 8 connection(s)
I20260929 17:53:00.752558 1292 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704375975685-392-0/minicluster-data/ts-2/data/info.pb
I20260929 17:53:00.756319 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskup3Soe/build/debug/bin/kudu as pid 1292
I20260929 17:53:00.756420 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790704375975685-392-0/minicluster-data/ts-2/wal/instance
I20260929 17:53:00.756665 392 external_mini_cluster.cc:1428] Setting key c26aa1ed8a7bee0f18b4a694c6ec0882
Sep 29 17:53:00 dist-test-slave-53nh krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704380, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 17:53:00.766739 1423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:00.754463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43611 (local address 127.0.98.62:37249)
0929 17:53:00.754679 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:00.754682 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:00.755474 (+ 792us) server_negotiation.cc:407] Connection header received
0929 17:53:00.756898 (+ 1424us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:00.756902 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:00.756964 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:00.757075 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:53:00.758466 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:00.759265 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:00.760398 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:00.760783 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:00.760896 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:00.763599 (+ 2703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:53:00.763602 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:53:00.763605 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:53:00.763628 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:53:00.764917 (+ 1289us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:53:00.765406 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:53:00.765408 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:53:00.765410 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:53:00.765483 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:53:00.765732 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:53:00.765734 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:53:00.765735 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:53:00.765899 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:53:00.765965 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0929 17:53:00.766496 (+ 531us) server_negotiation.cc:299] Negotiation successful
0929 17:53:00.766603 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":70,"threads_started":1}
I20260929 17:53:00.767518 1424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:00.754891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37249 (local address 127.0.98.3:43611)
0929 17:53:00.755324 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:00.755368 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 17:53:00.756651 (+ 1283us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:53:00.757256 (+ 605us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:53:00.757264 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:53:00.757712 (+ 448us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:53:00.758293 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:00.758313 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:00.759387 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:00.759390 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:00.760272 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:00.760291 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:00.760936 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:00.760939 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:00.761136 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:00.761933 (+ 797us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:53:00.761950 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:53:00.763439 (+ 1489us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:53:00.765039 (+ 1600us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:53:00.765045 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:53:00.765059 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:53:00.765326 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:53:00.765545 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:53:00.765547 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:53:00.765549 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:53:00.765671 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:53:00.765973 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:53:00.765978 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:53:00.766210 (+ 232us) client_negotiation.cc:769] Sending connection context
0929 17:53:00.766407 (+ 197us) client_negotiation.cc:240] Negotiation successful
0929 17:53:00.766671 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":109,"threads_started":1}
I20260929 17:53:00.768613 1421 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37249
I20260929 17:53:00.768846 1421 heartbeater.cc:464] Registering TS with master...
I20260929 17:53:00.769397 1421 heartbeater.cc:511] Master 127.0.98.62:37249 requested a full tablet report, sending...
I20260929 17:53:00.770452 961 ts_manager.cc:194] Registered new tserver with Master: 4cf0cb525aab4b19984f91feed8af780 (127.0.98.3:32967)
I20260929 17:53:00.771024 961 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:43611
I20260929 17:53:00.772593 1287 heartbeater.cc:503] Master 127.0.98.62:37249 was elected leader, sending a full tablet report...
I20260929 17:53:00.782274 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 17:53:00.793457 1423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:00.785782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50196 (local address 127.0.98.62:37249)
0929 17:53:00.785938 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:00.785944 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:00.786111 (+ 167us) server_negotiation.cc:407] Connection header received
0929 17:53:00.786242 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:00.786245 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:00.786286 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:00.786386 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:53:00.787101 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:00.787627 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:00.788456 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:00.788680 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:00.788767 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:00.790102 (+ 1335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:53:00.790106 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:53:00.790109 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:53:00.790137 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:53:00.791796 (+ 1659us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:53:00.792195 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:53:00.792199 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:53:00.792200 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:53:00.792287 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:53:00.792527 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:53:00.792530 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:53:00.792532 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:53:00.792750 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:53:00.792869 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0929 17:53:00.793116 (+ 247us) server_negotiation.cc:299] Negotiation successful
0929 17:53:00.793321 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 17:53:00.798269 961 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50196:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 17:53:00.801342 961 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 17:53:00.815945 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:00.810175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40803 (local address 127.0.0.1:38864)
0929 17:53:00.810658 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:00.810714 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0929 17:53:00.810857 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:53:00.811580 (+ 723us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:53:00.811586 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:53:00.811606 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:53:00.811735 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:00.811742 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:00.813166 (+ 1424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:00.813170 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:00.814269 (+ 1099us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:00.814279 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:00.815372 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:00.815376 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:00.815484 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:00.815526 (+ 42us) client_negotiation.cc:769] Sending connection context
0929 17:53:00.815620 (+ 94us) client_negotiation.cc:240] Negotiation successful
0929 17:53:00.815805 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":192,"threads_started":1}
I20260929 17:53:00.815933 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:00.810329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38864 (local address 127.0.98.1:40803)
0929 17:53:00.810775 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:00.810783 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:00.810809 (+ 26us) server_negotiation.cc:407] Connection header received
0929 17:53:00.811069 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:00.811076 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:00.811301 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:00.811456 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:53:00.811866 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:00.813050 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:00.814419 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:00.815251 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:00.815377 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:00.815582 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0929 17:53:00.815667 (+ 85us) server_negotiation.cc:299] Negotiation successful
0929 17:53:00.815767 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":107,"threads_started":1}
I20260929 17:53:00.816370 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:00.810576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42023 (local address 127.0.0.1:43054)
0929 17:53:00.811035 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:00.811047 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 17:53:00.811128 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:53:00.812241 (+ 1113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:53:00.812244 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:53:00.812264 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:53:00.812380 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:00.812388 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:00.813533 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:00.813536 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:00.814976 (+ 1440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:00.814984 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:00.816015 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:00.816019 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:00.816148 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:00.816168 (+ 20us) client_negotiation.cc:769] Sending connection context
0929 17:53:00.816219 (+ 51us) client_negotiation.cc:240] Negotiation successful
0929 17:53:00.816279 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":401,"mutex_wait_us":19,"thread_start_us":83,"threads_started":1}
I20260929 17:53:00.816550 1438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:00.811229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43054 (local address 127.0.98.2:42023)
0929 17:53:00.811747 (+ 518us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:00.811754 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:00.811776 (+ 22us) server_negotiation.cc:407] Connection header received
0929 17:53:00.811847 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:00.811852 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:00.812052 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:00.812249 (+ 197us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:53:00.812506 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:00.813436 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:00.815095 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:00.815848 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:00.816130 (+ 282us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:00.816269 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0929 17:53:00.816321 (+ 52us) server_negotiation.cc:299] Negotiation successful
0929 17:53:00.816407 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":420,"thread_start_us":93,"threads_started":1}
I20260929 17:53:00.816601 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:00.811045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:32967 (local address 127.0.0.1:59438)
0929 17:53:00.811371 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:00.811383 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 17:53:00.811459 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:53:00.812398 (+ 939us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:53:00.812401 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:53:00.812416 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:53:00.812510 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:00.812517 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:00.813713 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:00.813716 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:00.814811 (+ 1095us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:00.814821 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:00.815857 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:00.815861 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:00.815953 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:00.815975 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 17:53:00.816035 (+ 60us) client_negotiation.cc:240] Negotiation successful
0929 17:53:00.816494 (+ 459us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"mutex_wait_us":1,"thread_start_us":80,"threads_started":1}
I20260929 17:53:00.817085 1439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:00.811420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59438 (local address 127.0.98.3:32967)
0929 17:53:00.811829 (+ 409us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:00.811840 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:00.811860 (+ 20us) server_negotiation.cc:407] Connection header received
0929 17:53:00.811929 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:00.811933 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:00.812120 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:00.812260 (+ 140us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:53:00.812633 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:00.813567 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:00.814937 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:00.815583 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:00.815670 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:00.816734 (+ 1064us) server_negotiation.cc:1035] Waiting for connection context
0929 17:53:00.816783 (+ 49us) server_negotiation.cc:299] Negotiation successful
0929 17:53:00.816927 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320,"thread_start_us":115,"threads_started":1}
I20260929 17:53:00.819228 1087 tablet_service.cc:1467] Processing CreateTablet for tablet ed4ef71d6b174bd9879d8e72cf06e329 (DEFAULT_TABLE table=test-table [id=e812de1806c64fc0a65f10f6ecef7631]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 17:53:00.819262 1221 tablet_service.cc:1467] Processing CreateTablet for tablet ed4ef71d6b174bd9879d8e72cf06e329 (DEFAULT_TABLE table=test-table [id=e812de1806c64fc0a65f10f6ecef7631]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 17:53:00.819272 1355 tablet_service.cc:1467] Processing CreateTablet for tablet ed4ef71d6b174bd9879d8e72cf06e329 (DEFAULT_TABLE table=test-table [id=e812de1806c64fc0a65f10f6ecef7631]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 17:53:00.820200 1087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed4ef71d6b174bd9879d8e72cf06e329. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:53:00.820205 1355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed4ef71d6b174bd9879d8e72cf06e329. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:53:00.820210 1221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed4ef71d6b174bd9879d8e72cf06e329. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:53:00.827224 1440 tablet_bootstrap.cc:492] T ed4ef71d6b174bd9879d8e72cf06e329 P 23d4f46ee6a9414bb0b0408f1a4da527: Bootstrap starting.
I20260929 17:53:00.827689 1441 tablet_bootstrap.cc:492] T ed4ef71d6b174bd9879d8e72cf06e329 P 0f026596c3354a0ca27050ad958baed6: Bootstrap starting.
I20260929 17:53:00.828073 1442 tablet_bootstrap.cc:492] T ed4ef71d6b174bd9879d8e72cf06e329 P 4cf0cb525aab4b19984f91feed8af780: Bootstrap starting.
I20260929 17:53:00.830005 1442 tablet_bootstrap.cc:654] T ed4ef71d6b174bd9879d8e72cf06e329 P 4cf0cb525aab4b19984f91feed8af780: Neither blocks nor log segments found. Creating new log.
I20260929 17:53:00.830657 1441 tablet_bootstrap.cc:654] T ed4ef71d6b174bd9879d8e72cf06e329 P 0f026596c3354a0ca27050ad958baed6: Neither blocks nor log segments found. Creating new log.
I20260929 17:53:00.831331 1441 log.cc:824] T ed4ef71d6b174bd9879d8e72cf06e329 P 0f026596c3354a0ca27050ad958baed6: Log is configured to *not* fsync() on all Append() calls
I20260929 17:53:00.831350 1442 log.cc:824] T ed4ef71d6b174bd9879d8e72cf06e329 P 4cf0cb525aab4b19984f91feed8af780: Log is configured to *not* fsync() on all Append() calls
I20260929 17:53:00.832254 1440 tablet_bootstrap.cc:654] T ed4ef71d6b174bd9879d8e72cf06e329 P 23d4f46ee6a9414bb0b0408f1a4da527: Neither blocks nor log segments found. Creating new log.
I20260929 17:53:00.833089 1440 log.cc:824] T ed4ef71d6b174bd9879d8e72cf06e329 P 23d4f46ee6a9414bb0b0408f1a4da527: Log is configured to *not* fsync() on all Append() calls
I20260929 17:53:00.833557 1441 tablet_bootstrap.cc:492] T ed4ef71d6b174bd9879d8e72cf06e329 P 0f026596c3354a0ca27050ad958baed6: No bootstrap required, opened a new log
I20260929 17:53:00.833762 1441 ts_tablet_manager.cc:1397] T ed4ef71d6b174bd9879d8e72cf06e329 P 0f026596c3354a0ca27050ad958baed6: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 17:53:00.834277 1442 tablet_bootstrap.cc:492] T ed4ef71d6b174bd9879d8e72cf06e329 P 4cf0cb525aab4b19984f91feed8af780: No bootstrap required, opened a new log
I20260929 17:53:00.834457 1442 ts_tablet_manager.cc:1397] T ed4ef71d6b174bd9879d8e72cf06e329 P 4cf0cb525aab4b19984f91feed8af780: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260929 17:53:00.835160 1440 tablet_bootstrap.cc:492] T ed4ef71d6b174bd9879d8e72cf06e329 P 23d4f46ee6a9414bb0b0408f1a4da527: No bootstrap required, opened a new log
I20260929 17:53:00.835354 1440 ts_tablet_manager.cc:1397] T ed4ef71d6b174bd9879d8e72cf06e329 P 23d4f46ee6a9414bb0b0408f1a4da527: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20260929 17:53:00.837383 1442 raft_consensus.cc:348] T ed4ef71d6b174bd9879d8e72cf06e329 P 4cf0cb525aab4b19984f91feed8af780 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23d4f46ee6a9414bb0b0408f1a4da527" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40803 } } peers { permanent_uuid: "4cf0cb525aab4b19984f91feed8af780" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32967 } } peers { permanent_uuid: "0f026596c3354a0ca27050ad958baed6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42023 } }
I20260929 17:53:00.837620 1442 raft_consensus.cc:374] T ed4ef71d6b174bd9879d8e72cf06e329 P 4cf0cb525aab4b19984f91feed8af780 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:53:00.837671 1442 raft_consensus.cc:729] T ed4ef71d6b174bd9879d8e72cf06e329 P 4cf0cb525aab4b19984f91feed8af780 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cf0cb525aab4b19984f91feed8af780, State: Initialized, Role: FOLLOWER
I20260929 17:53:00.838102 1442 consensus_queue.cc:260] T ed4ef71d6b174bd9879d8e72cf06e329 P 4cf0cb525aab4b19984f91feed8af780 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23d4f46ee6a9414bb0b0408f1a4da527" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40803 } } peers { permanent_uuid: "4cf0cb525aab4b19984f91feed8af780" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32967 } } peers { permanent_uuid: "0f026596c3354a0ca27050ad958baed6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42023 } }
I20260929 17:53:00.838773 1421 heartbeater.cc:503] Master 127.0.98.62:37249 was elected leader, sending a full tablet report...
I20260929 17:53:00.838853 1441 raft_consensus.cc:348] T ed4ef71d6b174bd9879d8e72cf06e329 P 0f026596c3354a0ca27050ad958baed6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23d4f46ee6a9414bb0b0408f1a4da527" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40803 } } peers { permanent_uuid: "4cf0cb525aab4b19984f91feed8af780" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32967 } } peers { permanent_uuid: "0f026596c3354a0ca27050ad958baed6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42023 } }
I20260929 17:53:00.839114 1441 raft_consensus.cc:374] T ed4ef71d6b174bd9879d8e72cf06e329 P 0f026596c3354a0ca27050ad958baed6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:53:00.839128 1442 ts_tablet_manager.cc:1428] T ed4ef71d6b174bd9879d8e72cf06e329 P 4cf0cb525aab4b19984f91feed8af780: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260929 17:53:00.839166 1441 raft_consensus.cc:729] T ed4ef71d6b174bd9879d8e72cf06e329 P 0f026596c3354a0ca27050ad958baed6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f026596c3354a0ca27050ad958baed6, State: Initialized, Role: FOLLOWER
I20260929 17:53:00.839623 1441 consensus_queue.cc:260] T ed4ef71d6b174bd9879d8e72cf06e329 P 0f026596c3354a0ca27050ad958baed6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23d4f46ee6a9414bb0b0408f1a4da527" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40803 } } peers { permanent_uuid: "4cf0cb525aab4b19984f91feed8af780" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32967 } } peers { permanent_uuid: "0f026596c3354a0ca27050ad958baed6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42023 } }
I20260929 17:53:00.840001 1440 raft_consensus.cc:348] T ed4ef71d6b174bd9879d8e72cf06e329 P 23d4f46ee6a9414bb0b0408f1a4da527 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23d4f46ee6a9414bb0b0408f1a4da527" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40803 } } peers { permanent_uuid: "4cf0cb525aab4b19984f91feed8af780" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32967 } } peers { permanent_uuid: "0f026596c3354a0ca27050ad958baed6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42023 } }
I20260929 17:53:00.840281 1440 raft_consensus.cc:374] T ed4ef71d6b174bd9879d8e72cf06e329 P 23d4f46ee6a9414bb0b0408f1a4da527 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:53:00.840341 1440 raft_consensus.cc:729] T ed4ef71d6b174bd9879d8e72cf06e329 P 23d4f46ee6a9414bb0b0408f1a4da527 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23d4f46ee6a9414bb0b0408f1a4da527, State: Initialized, Role: FOLLOWER
I20260929 17:53:00.840641 1441 ts_tablet_manager.cc:1428] T ed4ef71d6b174bd9879d8e72cf06e329 P 0f026596c3354a0ca27050ad958baed6: Time spent starting tablet: real 0.007s user 0.001s sys 0.005s
I20260929 17:53:00.840819 1440 consensus_queue.cc:260] T ed4ef71d6b174bd9879d8e72cf06e329 P 23d4f46ee6a9414bb0b0408f1a4da527 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23d4f46ee6a9414bb0b0408f1a4da527" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40803 } } peers { permanent_uuid: "4cf0cb525aab4b19984f91feed8af780" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32967 } } peers { permanent_uuid: "0f026596c3354a0ca27050ad958baed6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42023 } }
I20260929 17:53:00.842491 1440 ts_tablet_manager.cc:1428] T ed4ef71d6b174bd9879d8e72cf06e329 P 23d4f46ee6a9414bb0b0408f1a4da527: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260929 17:53:00.857913 1447 raft_consensus.cc:482] T ed4ef71d6b174bd9879d8e72cf06e329 P 0f026596c3354a0ca27050ad958baed6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 17:53:00.858121 1447 raft_consensus.cc:504] T ed4ef71d6b174bd9879d8e72cf06e329 P 0f026596c3354a0ca27050ad958baed6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23d4f46ee6a9414bb0b0408f1a4da527" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40803 } } peers { permanent_uuid: "4cf0cb525aab4b19984f91feed8af780" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32967 } } peers { permanent_uuid: "0f026596c3354a0ca27050ad958baed6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42023 } }
I20260929 17:53:00.859249 1447 leader_election.cc:288] T ed4ef71d6b174bd9879d8e72cf06e329 P 0f026596c3354a0ca27050ad958baed6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23d4f46ee6a9414bb0b0408f1a4da527 (127.0.98.1:40803), 4cf0cb525aab4b19984f91feed8af780 (127.0.98.3:32967)
I20260929 17:53:00.864259 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:00.859605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41657 (local address 127.0.98.1:40803)
0929 17:53:00.859752 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:00.859757 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:00.859967 (+ 210us) server_negotiation.cc:407] Connection header received
0929 17:53:00.860178 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:00.860182 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:00.860238 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:00.860287 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:53:00.860701 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:00.861505 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:00.863048 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:00.863701 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:00.863811 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:00.863889 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0929 17:53:00.864056 (+ 167us) server_negotiation.cc:299] Negotiation successful
0929 17:53:00.864124 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260929 17:53:00.864259 1439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:00.860059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59855 (local address 127.0.98.3:32967)
0929 17:53:00.860171 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:00.860175 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:00.860401 (+ 226us) server_negotiation.cc:407] Connection header received
0929 17:53:00.860583 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:00.860585 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:00.860628 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:00.860709 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:53:00.860965 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:00.861675 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:00.863149 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:00.863752 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:00.863836 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:00.863903 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0929 17:53:00.864059 (+ 156us) server_negotiation.cc:299] Negotiation successful
0929 17:53:00.864124 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 17:53:00.864328 1450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:00.859951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:32967 (local address 127.0.98.2:59855)
0929 17:53:00.860322 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:00.860334 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 17:53:00.860465 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:53:00.860735 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:53:00.860737 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:53:00.860753 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:53:00.860841 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:00.860868 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:00.861767 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:00.861770 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:00.863041 (+ 1271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:00.863049 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:00.863836 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:00.863839 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:00.863965 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:00.863982 (+ 17us) client_negotiation.cc:769] Sending connection context
0929 17:53:00.864044 (+ 62us) client_negotiation.cc:240] Negotiation successful
0929 17:53:00.864126 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"mutex_wait_us":41,"thread_start_us":104,"threads_started":1}
I20260929 17:53:00.864504 1449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:00.859521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40803 (local address 127.0.98.2:41657)
0929 17:53:00.859883 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:00.859898 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 17:53:00.860048 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:53:00.860465 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:53:00.860468 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:53:00.860481 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:53:00.860601 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:00.860607 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:00.861600 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:00.861603 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:00.862915 (+ 1312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:00.862926 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:00.863809 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:00.863812 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:00.863927 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:00.863956 (+ 29us) client_negotiation.cc:769] Sending connection context
0929 17:53:00.864036 (+ 80us) client_negotiation.cc:240] Negotiation successful
0929 17:53:00.864126 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"mutex_wait_us":26,"thread_start_us":140,"threads_started":1}
I20260929 17:53:00.864946 1375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed4ef71d6b174bd9879d8e72cf06e329" candidate_uuid: "0f026596c3354a0ca27050ad958baed6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cf0cb525aab4b19984f91feed8af780" is_pre_election: true
I20260929 17:53:00.865072 1107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed4ef71d6b174bd9879d8e72cf06e329" candidate_uuid: "0f026596c3354a0ca27050ad958baed6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23d4f46ee6a9414bb0b0408f1a4da527" is_pre_election: true
I20260929 17:53:00.865320 1107 raft_consensus.cc:2454] T ed4ef71d6b174bd9879d8e72cf06e329 P 23d4f46ee6a9414bb0b0408f1a4da527 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0f026596c3354a0ca27050ad958baed6 in term 0.
I20260929 17:53:00.865320 1375 raft_consensus.cc:2454] T ed4ef71d6b174bd9879d8e72cf06e329 P 4cf0cb525aab4b19984f91feed8af780 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0f026596c3354a0ca27050ad958baed6 in term 0.
I20260929 17:53:00.865790 1176 leader_election.cc:302] T ed4ef71d6b174bd9879d8e72cf06e329 P 0f026596c3354a0ca27050ad958baed6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f026596c3354a0ca27050ad958baed6, 4cf0cb525aab4b19984f91feed8af780; no voters:
I20260929 17:53:00.866027 1447 raft_consensus.cc:2781] T ed4ef71d6b174bd9879d8e72cf06e329 P 0f026596c3354a0ca27050ad958baed6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 17:53:00.866107 1447 raft_consensus.cc:482] T ed4ef71d6b174bd9879d8e72cf06e329 P 0f026596c3354a0ca27050ad958baed6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 17:53:00.866153 1447 raft_consensus.cc:3037] T ed4ef71d6b174bd9879d8e72cf06e329 P 0f026596c3354a0ca27050ad958baed6 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:53:00.886524 1447 raft_consensus.cc:504] T ed4ef71d6b174bd9879d8e72cf06e329 P 0f026596c3354a0ca27050ad958baed6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23d4f46ee6a9414bb0b0408f1a4da527" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40803 } } peers { permanent_uuid: "4cf0cb525aab4b19984f91feed8af780" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32967 } } peers { permanent_uuid: "0f026596c3354a0ca27050ad958baed6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42023 } }
I20260929 17:53:00.887017 1447 leader_election.cc:288] T ed4ef71d6b174bd9879d8e72cf06e329 P 0f026596c3354a0ca27050ad958baed6 [CANDIDATE]: Term 1 election: Requested vote from peers 23d4f46ee6a9414bb0b0408f1a4da527 (127.0.98.1:40803), 4cf0cb525aab4b19984f91feed8af780 (127.0.98.3:32967)
I20260929 17:53:00.887537 1107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed4ef71d6b174bd9879d8e72cf06e329" candidate_uuid: "0f026596c3354a0ca27050ad958baed6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23d4f46ee6a9414bb0b0408f1a4da527"
I20260929 17:53:00.887593 1375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed4ef71d6b174bd9879d8e72cf06e329" candidate_uuid: "0f026596c3354a0ca27050ad958baed6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cf0cb525aab4b19984f91feed8af780"
I20260929 17:53:00.887742 1375 raft_consensus.cc:3037] T ed4ef71d6b174bd9879d8e72cf06e329 P 4cf0cb525aab4b19984f91feed8af780 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:53:00.887702 1107 raft_consensus.cc:3037] T ed4ef71d6b174bd9879d8e72cf06e329 P 23d4f46ee6a9414bb0b0408f1a4da527 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:53:00.889278 1375 raft_consensus.cc:2454] T ed4ef71d6b174bd9879d8e72cf06e329 P 4cf0cb525aab4b19984f91feed8af780 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f026596c3354a0ca27050ad958baed6 in term 1.
I20260929 17:53:00.889286 1107 raft_consensus.cc:2454] T ed4ef71d6b174bd9879d8e72cf06e329 P 23d4f46ee6a9414bb0b0408f1a4da527 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f026596c3354a0ca27050ad958baed6 in term 1.
I20260929 17:53:00.889652 1176 leader_election.cc:302] T ed4ef71d6b174bd9879d8e72cf06e329 P 0f026596c3354a0ca27050ad958baed6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f026596c3354a0ca27050ad958baed6, 23d4f46ee6a9414bb0b0408f1a4da527; no voters:
I20260929 17:53:00.889869 1447 raft_consensus.cc:2781] T ed4ef71d6b174bd9879d8e72cf06e329 P 0f026596c3354a0ca27050ad958baed6 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 17:53:00.890137 1447 raft_consensus.cc:686] T ed4ef71d6b174bd9879d8e72cf06e329 P 0f026596c3354a0ca27050ad958baed6 [term 1 LEADER]: Becoming Leader. State: Replica: 0f026596c3354a0ca27050ad958baed6, State: Running, Role: LEADER
I20260929 17:53:00.890594 1447 consensus_queue.cc:237] T ed4ef71d6b174bd9879d8e72cf06e329 P 0f026596c3354a0ca27050ad958baed6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23d4f46ee6a9414bb0b0408f1a4da527" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40803 } } peers { permanent_uuid: "4cf0cb525aab4b19984f91feed8af780" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32967 } } peers { permanent_uuid: "0f026596c3354a0ca27050ad958baed6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42023 } }
I20260929 17:53:00.893887 961 catalog_manager.cc:5495] T ed4ef71d6b174bd9879d8e72cf06e329 P 0f026596c3354a0ca27050ad958baed6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0f026596c3354a0ca27050ad958baed6 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "0f026596c3354a0ca27050ad958baed6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23d4f46ee6a9414bb0b0408f1a4da527" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40803 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4cf0cb525aab4b19984f91feed8af780" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32967 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0f026596c3354a0ca27050ad958baed6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42023 } health_report { overall_health: HEALTHY } } }
Sep 29 17:53:00 dist-test-slave-53nh krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704379, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260929 17:53:00.919059 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:00.909897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38872 (local address 127.0.98.1:40803)
0929 17:53:00.910026 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:00.910031 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:00.910614 (+ 583us) server_negotiation.cc:407] Connection header received
0929 17:53:00.910749 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:00.910752 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:00.910799 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:00.910866 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:53:00.911530 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:00.912066 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:00.912892 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:00.913149 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:00.913248 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:00.915583 (+ 2335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:53:00.915597 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:53:00.915613 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:53:00.915637 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:53:00.917361 (+ 1724us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:53:00.917788 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:53:00.917800 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:53:00.917801 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:53:00.917892 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:53:00.918137 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:53:00.918140 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:53:00.918142 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:53:00.918404 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:53:00.918611 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0929 17:53:00.918778 (+ 167us) server_negotiation.cc:299] Negotiation successful
0929 17:53:00.918904 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
Sep 29 17:53:00 dist-test-slave-53nh krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704380, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 17:53:00 dist-test-slave-53nh krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704380, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260929 17:53:00.993556 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:00.985290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38876 (local address 127.0.98.1:40803)
0929 17:53:00.985418 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:00.985422 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:00.985595 (+ 173us) server_negotiation.cc:407] Connection header received
0929 17:53:00.985723 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:00.985727 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:00.985774 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:00.985842 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:53:00.986540 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:00.987064 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:00.987877 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:00.988104 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:00.988188 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:00.990678 (+ 2490us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:53:00.990682 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:53:00.990684 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:53:00.990708 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:53:00.992031 (+ 1323us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:53:00.992404 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:53:00.992407 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:53:00.992409 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:53:00.992481 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:53:00.992705 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:53:00.992709 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:53:00.992710 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:53:00.992874 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:53:00.992984 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0929 17:53:00.993286 (+ 302us) server_negotiation.cc:299] Negotiation successful
0929 17:53:00.993400 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
W20260929 17:53:01.002923 1288 tablet.cc:2369] T ed4ef71d6b174bd9879d8e72cf06e329 P 0f026596c3354a0ca27050ad958baed6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 17:53:01.003552 1422 tablet.cc:2369] T ed4ef71d6b174bd9879d8e72cf06e329 P 4cf0cb525aab4b19984f91feed8af780: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 29 17:53:00 dist-test-slave-53nh krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704380, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
W20260929 17:53:01.034566 1154 tablet.cc:2369] T ed4ef71d6b174bd9879d8e72cf06e329 P 23d4f46ee6a9414bb0b0408f1a4da527: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 29 17:53:01 dist-test-slave-53nh krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704380, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260929 17:53:01.102978 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:01.088654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38880 (local address 127.0.98.1:40803)
0929 17:53:01.088778 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:01.088783 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:01.088967 (+ 184us) server_negotiation.cc:407] Connection header received
0929 17:53:01.089143 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:01.089146 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:01.089191 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:01.089255 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:53:01.089973 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:01.090506 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:01.091455 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:01.091667 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:01.091753 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:01.099842 (+ 8089us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:53:01.099846 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:53:01.099849 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:53:01.099878 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:53:01.101517 (+ 1639us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:53:01.101968 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:53:01.101971 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:53:01.101972 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:53:01.102052 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:53:01.102316 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:53:01.102319 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:53:01.102320 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:53:01.102485 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:53:01.102581 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0929 17:53:01.102755 (+ 174us) server_negotiation.cc:299] Negotiation successful
0929 17:53:01.102858 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
W20260929 17:53:01.104352 1067 tablet_service.cc:2908] Rejecting scan request for tablet ed4ef71d6b174bd9879d8e72cf06e329: Uninitialized: clean time has not yet been initialized
I20260929 17:53:01.106369 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskup3Soe/build/debug/bin/kudu with pid 1024
I20260929 17:53:01.111620 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskup3Soe/build/debug/bin/kudu with pid 1158
I20260929 17:53:01.116245 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskup3Soe/build/debug/bin/kudu with pid 1292
I20260929 17:53:01.120873 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskup3Soe/build/debug/bin/kudu with pid 930
2026-09-29T17:53:01Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (3094 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704375975685-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 17:53:02 dist-test-slave-53nh krb5kdc[1465](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 17:53:02 dist-test-slave-53nh krb5kdc[1465](info): set up 2 sockets
Sep 29 17:53:02 dist-test-slave-53nh krb5kdc[1465](info): commencing operation
krb5kdc: starting...
W20260929 17:53:03.481114 392 mini_kdc.cc:121] Time spent starting KDC: real 2.339s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260929 17:53:03.647886 392 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.167s user 0.001s sys 0.001s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
W20260929 17:53:03.824712 392 mini_kdc.cc:253] Time spent creating user principal test-user: real 0.177s user 0.000s sys 0.002s
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
W20260929 17:53:03.988777 392 mini_kdc.cc:253] Time spent creating user principal joe-interloper: real 0.164s user 0.002s sys 0.000s
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704375975685-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704375975685-392-0/krb5kdc/test-user.keytab.
W20260929 17:53:04.479489 392 mini_kdc.cc:302] Time spent creating keytab for test-user: real 0.491s user 0.000s sys 0.002s
Sep 29 17:53:04 dist-test-slave-53nh krb5kdc[1465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704384, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T17:53:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T17:53:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704375975685-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704375975685-392-0/krb5kdc/kudu_127.0.98.62.keytab.
W20260929 17:53:05.327881 392 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.98.62: real 0.827s user 0.000s sys 0.003s
I20260929 17:53:05.328420 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskup3Soe/build/debug/bin/kudu
/tmp/dist-test-taskup3Soe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704375975685-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704375975685-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704375975685-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704375975685-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42603
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704375975685-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42603
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704375975685-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704375975685-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704375975685-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:53:05.436362 1478 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:53:05.436621 1478 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:53:05.436664 1478 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:53:05.440666 1478 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 17:53:05.440728 1478 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260929 17:53:05.440744 1478 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:53:05.440761 1478 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:53:05.440775 1478 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 17:53:05.440789 1478 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 17:53:05.446126 1478 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704375975685-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704375975685-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42603
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704375975685-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42603
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704375975685-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1478
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704375975685-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:53:05.447142 1478 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:53:05.448137 1478 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:53:05.456116 1486 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:53:05.456131 1484 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:53:05.456137 1483 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:53:05.456547 1478 server_base.cc:1034] running on GCE node
I20260929 17:53:05.457072 1478 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:53:05.458129 1478 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:53:05.459306 1478 hybrid_clock.cc:648] HybridClock initialized: now 1790704385459290 us; error 27 us; skew 500 ppm
Sep 29 17:53:05 dist-test-slave-53nh krb5kdc[1465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704385, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:53:05.462007 1478 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 17:53:05.462889 1478 webserver.cc:460] Webserver started at http://127.0.98.62:41321/ using document root <none> and password file <none>
I20260929 17:53:05.463423 1478 fs_manager.cc:359] Metadata directory not provided
I20260929 17:53:05.463480 1478 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:53:05.463663 1478 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:53:05.465780 1478 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704375975685-392-0/minicluster-data/master-0/data/instance:
uuid: "bf0bd8ed30e946f49cc31cdf2f3f5424"
format_stamp: "Formatted at 2026-09-29 17:53:05 on dist-test-slave-53nh"
server_key: "390b6ace163e8365c2c6521dc298f0d8"
server_key_iv: "a193f56699cbf1937731db3852f95981"
server_key_version: "encryptionkey@0"
I20260929 17:53:05.466370 1478 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704375975685-392-0/minicluster-data/master-0/wal/instance:
uuid: "bf0bd8ed30e946f49cc31cdf2f3f5424"
format_stamp: "Formatted at 2026-09-29 17:53:05 on dist-test-slave-53nh"
server_key: "390b6ace163e8365c2c6521dc298f0d8"
server_key_iv: "a193f56699cbf1937731db3852f95981"
server_key_version: "encryptionkey@0"
I20260929 17:53:05.470124 1478 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 17:53:05.472658 1493 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:53:05.473942 1478 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 17:53:05.474069 1478 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704375975685-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704375975685-392-0/minicluster-data/master-0/wal
uuid: "bf0bd8ed30e946f49cc31cdf2f3f5424"
format_stamp: "Formatted at 2026-09-29 17:53:05 on dist-test-slave-53nh"
server_key: "390b6ace163e8365c2c6521dc298f0d8"
server_key_iv: "a193f56699cbf1937731db3852f95981"
server_key_version: "encryptionkey@0"
I20260929 17:53:05.474182 1478 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704375975685-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704375975685-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704375975685-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:53:05.493815 1478 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:53:05.496822 1478 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:53:05.497059 1478 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:53:05.505309 1478 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42603
I20260929 17:53:05.505337 1555 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42603 every 8 connection(s)
I20260929 17:53:05.506311 1478 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704375975685-392-0/minicluster-data/master-0/data/info.pb
I20260929 17:53:05.509245 1556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:53:05.514429 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskup3Soe/build/debug/bin/kudu as pid 1478
I20260929 17:53:05.514535 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704375975685-392-0/minicluster-data/master-0/wal/instance
I20260929 17:53:05.514756 392 external_mini_cluster.cc:1428] Setting key 132140e43c14a94fe8ec7837e8b2daf2
I20260929 17:53:05.514894 1556 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf0bd8ed30e946f49cc31cdf2f3f5424: Bootstrap starting.
I20260929 17:53:05.517588 1556 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bf0bd8ed30e946f49cc31cdf2f3f5424: Neither blocks nor log segments found. Creating new log.
I20260929 17:53:05.518465 1556 log.cc:824] T 00000000000000000000000000000000 P bf0bd8ed30e946f49cc31cdf2f3f5424: Log is configured to *not* fsync() on all Append() calls
I20260929 17:53:05.520330 1556 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf0bd8ed30e946f49cc31cdf2f3f5424: No bootstrap required, opened a new log
Sep 29 17:53:05 dist-test-slave-53nh krb5kdc[1465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704384, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 17:53:05.523977 1556 raft_consensus.cc:348] T 00000000000000000000000000000000 P bf0bd8ed30e946f49cc31cdf2f3f5424 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf0bd8ed30e946f49cc31cdf2f3f5424" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42603 } }
I20260929 17:53:05.524211 1556 raft_consensus.cc:374] T 00000000000000000000000000000000 P bf0bd8ed30e946f49cc31cdf2f3f5424 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:53:05.524247 1556 raft_consensus.cc:729] T 00000000000000000000000000000000 P bf0bd8ed30e946f49cc31cdf2f3f5424 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf0bd8ed30e946f49cc31cdf2f3f5424, State: Initialized, Role: FOLLOWER
I20260929 17:53:05.524722 1556 consensus_queue.cc:260] T 00000000000000000000000000000000 P bf0bd8ed30e946f49cc31cdf2f3f5424 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf0bd8ed30e946f49cc31cdf2f3f5424" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42603 } }
I20260929 17:53:05.524878 1556 raft_consensus.cc:388] T 00000000000000000000000000000000 P bf0bd8ed30e946f49cc31cdf2f3f5424 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 17:53:05.524928 1556 raft_consensus.cc:482] T 00000000000000000000000000000000 P bf0bd8ed30e946f49cc31cdf2f3f5424 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 17:53:05.525022 1556 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bf0bd8ed30e946f49cc31cdf2f3f5424 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:53:05.526082 1556 raft_consensus.cc:504] T 00000000000000000000000000000000 P bf0bd8ed30e946f49cc31cdf2f3f5424 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf0bd8ed30e946f49cc31cdf2f3f5424" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42603 } }
I20260929 17:53:05.526418 1556 leader_election.cc:302] T 00000000000000000000000000000000 P bf0bd8ed30e946f49cc31cdf2f3f5424 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bf0bd8ed30e946f49cc31cdf2f3f5424; no voters:
I20260929 17:53:05.526739 1556 leader_election.cc:288] T 00000000000000000000000000000000 P bf0bd8ed30e946f49cc31cdf2f3f5424 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 17:53:05.526849 1561 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bf0bd8ed30e946f49cc31cdf2f3f5424 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 17:53:05.527060 1561 raft_consensus.cc:686] T 00000000000000000000000000000000 P bf0bd8ed30e946f49cc31cdf2f3f5424 [term 1 LEADER]: Becoming Leader. State: Replica: bf0bd8ed30e946f49cc31cdf2f3f5424, State: Running, Role: LEADER
I20260929 17:53:05.527405 1561 consensus_queue.cc:237] T 00000000000000000000000000000000 P bf0bd8ed30e946f49cc31cdf2f3f5424 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf0bd8ed30e946f49cc31cdf2f3f5424" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42603 } }
I20260929 17:53:05.527957 1556 sys_catalog.cc:564] T 00000000000000000000000000000000 P bf0bd8ed30e946f49cc31cdf2f3f5424 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 17:53:05.529075 1562 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf0bd8ed30e946f49cc31cdf2f3f5424 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bf0bd8ed30e946f49cc31cdf2f3f5424. Latest consensus state: current_term: 1 leader_uuid: "bf0bd8ed30e946f49cc31cdf2f3f5424" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf0bd8ed30e946f49cc31cdf2f3f5424" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42603 } } }
I20260929 17:53:05.529168 1563 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf0bd8ed30e946f49cc31cdf2f3f5424 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bf0bd8ed30e946f49cc31cdf2f3f5424" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf0bd8ed30e946f49cc31cdf2f3f5424" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42603 } } }
I20260929 17:53:05.529286 1562 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf0bd8ed30e946f49cc31cdf2f3f5424 [sys.catalog]: This master's current role is: LEADER
I20260929 17:53:05.529321 1563 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf0bd8ed30e946f49cc31cdf2f3f5424 [sys.catalog]: This master's current role is: LEADER
I20260929 17:53:05.529456 1559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:05.516120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36870 (local address 127.0.98.62:42603)
0929 17:53:05.516563 (+ 443us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:05.516582 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:05.516624 (+ 42us) server_negotiation.cc:407] Connection header received
0929 17:53:05.517464 (+ 840us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:05.517488 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:05.517971 (+ 483us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:05.518334 (+ 363us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:53:05.519044 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:05.520055 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:05.520901 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:05.521426 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:05.521570 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:05.524012 (+ 2442us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:53:05.524017 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:53:05.524031 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:53:05.524057 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:53:05.525933 (+ 1876us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:53:05.526400 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:53:05.526407 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:53:05.526412 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:53:05.526550 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:53:05.526901 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:53:05.526905 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:53:05.526907 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:53:05.527379 (+ 472us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:53:05.527555 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0929 17:53:05.528133 (+ 578us) server_negotiation.cc:299] Negotiation successful
0929 17:53:05.528432 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":303,"thread_start_us":125,"threads_started":1}
I20260929 17:53:05.529771 1572 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 17:53:05.532727 1572 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 17:53:05.537873 1572 catalog_manager.cc:1347] Generated new cluster ID: 4388a336cdeb4c5cba8284ab689303a1
I20260929 17:53:05.538019 1572 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 17:53:05.556501 1572 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 17:53:05.557948 1572 catalog_manager.cc:1504] Loading token signing keys...
I20260929 17:53:05.564966 1572 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bf0bd8ed30e946f49cc31cdf2f3f5424: Generated new TSK 0
I20260929 17:53:05.565698 1572 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704375975685-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704375975685-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260929 17:53:05.593900 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskup3Soe/build/debug/bin/kudu
/tmp/dist-test-taskup3Soe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704375975685-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704375975685-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704375975685-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704375975685-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42603
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704375975685-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704375975685-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704375975685-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704375975685-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:53:05.702224 1582 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:53:05.702459 1582 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:53:05.702500 1582 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:53:05.706467 1582 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:53:05.706519 1582 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:53:05.706555 1582 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 17:53:05.706593 1582 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 17:53:05.711555 1582 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704375975685-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704375975685-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704375975685-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704375975685-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:42603
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1582
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704375975685-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:53:05.712467 1582 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:53:05.713423 1582 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:53:05.722376 1587 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:53:05.722388 1590 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:53:05.722528 1582 server_base.cc:1034] running on GCE node
W20260929 17:53:05.722400 1588 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:53:05.723088 1582 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:53:05.723683 1582 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:53:05.724857 1582 hybrid_clock.cc:648] HybridClock initialized: now 1790704385724846 us; error 42 us; skew 500 ppm
Sep 29 17:53:05 dist-test-slave-53nh krb5kdc[1465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704385, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:53:05.727593 1582 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260929 17:53:05.728492 1582 webserver.cc:460] Webserver started at http://127.0.98.1:38459/ using document root <none> and password file <none>
I20260929 17:53:05.729072 1582 fs_manager.cc:359] Metadata directory not provided
I20260929 17:53:05.729149 1582 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:53:05.729359 1582 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:53:05.731382 1582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704375975685-392-0/minicluster-data/ts-0/data/instance:
uuid: "0ff660fa6d574e1a83e0ccc53672f485"
format_stamp: "Formatted at 2026-09-29 17:53:05 on dist-test-slave-53nh"
server_key: "8f9a1ee7d10313697511bc4312e76a51"
server_key_iv: "07b95934a11b3d03535f9b01cbbe421e"
server_key_version: "encryptionkey@0"
I20260929 17:53:05.731879 1582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704375975685-392-0/minicluster-data/ts-0/wal/instance:
uuid: "0ff660fa6d574e1a83e0ccc53672f485"
format_stamp: "Formatted at 2026-09-29 17:53:05 on dist-test-slave-53nh"
server_key: "8f9a1ee7d10313697511bc4312e76a51"
server_key_iv: "07b95934a11b3d03535f9b01cbbe421e"
server_key_version: "encryptionkey@0"
I20260929 17:53:05.735754 1582 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260929 17:53:05.738014 1597 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:53:05.739173 1582 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 17:53:05.739305 1582 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704375975685-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704375975685-392-0/minicluster-data/ts-0/wal
uuid: "0ff660fa6d574e1a83e0ccc53672f485"
format_stamp: "Formatted at 2026-09-29 17:53:05 on dist-test-slave-53nh"
server_key: "8f9a1ee7d10313697511bc4312e76a51"
server_key_iv: "07b95934a11b3d03535f9b01cbbe421e"
server_key_version: "encryptionkey@0"
I20260929 17:53:05.739406 1582 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704375975685-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704375975685-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704375975685-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:53:05.765909 1582 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:53:05.768668 1582 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:53:05.768837 1582 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:53:05.770264 1582 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:53:05.770314 1582 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:53:05.770367 1582 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:53:05.770396 1582 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 17:53:05 dist-test-slave-53nh krb5kdc[1465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704385, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 17:53:05.780006 1582 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40623
I20260929 17:53:05.780040 1712 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40623 every 8 connection(s)
I20260929 17:53:05.780973 1582 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704375975685-392-0/minicluster-data/ts-0/data/info.pb
I20260929 17:53:05.782670 1559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:05.771013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47117 (local address 127.0.98.62:42603)
0929 17:53:05.771158 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:05.771162 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:05.771948 (+ 786us) server_negotiation.cc:407] Connection header received
0929 17:53:05.773048 (+ 1100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:05.773051 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:05.773108 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:05.773163 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:53:05.774507 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:05.775194 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:05.776280 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:05.776563 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:05.776654 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:05.779070 (+ 2416us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:53:05.779073 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:53:05.779075 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:53:05.779098 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:53:05.780589 (+ 1491us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:53:05.781077 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:53:05.781081 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:53:05.781083 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:53:05.781166 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:53:05.781499 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:53:05.781503 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:53:05.781504 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:53:05.781792 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:53:05.781884 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0929 17:53:05.782381 (+ 497us) server_negotiation.cc:299] Negotiation successful
0929 17:53:05.782507 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 17:53:05.783432 1606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:05.771369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42603 (local address 127.0.98.1:47117)
0929 17:53:05.771792 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:05.771841 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 17:53:05.772783 (+ 942us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:53:05.773483 (+ 700us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:53:05.773489 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:53:05.773799 (+ 310us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:53:05.774375 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:05.774390 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:05.775311 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:05.775315 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:05.776172 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:05.776180 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:05.776709 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:05.776713 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:05.776871 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:05.777429 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:53:05.777444 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:53:05.778916 (+ 1472us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:53:05.780693 (+ 1777us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:53:05.780698 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:53:05.780713 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:53:05.780954 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:53:05.781246 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:53:05.781249 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:53:05.781251 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:53:05.781391 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:53:05.781879 (+ 488us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:53:05.781886 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:53:05.782154 (+ 268us) client_negotiation.cc:769] Sending connection context
0929 17:53:05.782323 (+ 169us) client_negotiation.cc:240] Negotiation successful
0929 17:53:05.782638 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":101,"threads_started":1}
I20260929 17:53:05.784623 1713 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42603
I20260929 17:53:05.784754 1713 heartbeater.cc:464] Registering TS with master...
I20260929 17:53:05.785341 1713 heartbeater.cc:511] Master 127.0.98.62:42603 requested a full tablet report, sending...
I20260929 17:53:05.786859 1509 ts_manager.cc:194] Registered new tserver with Master: 0ff660fa6d574e1a83e0ccc53672f485 (127.0.98.1:40623)
I20260929 17:53:05.788228 1509 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:47117
I20260929 17:53:05.789573 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskup3Soe/build/debug/bin/kudu as pid 1582
I20260929 17:53:05.789682 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704375975685-392-0/minicluster-data/ts-0/wal/instance
I20260929 17:53:05.789963 392 external_mini_cluster.cc:1428] Setting key a5b034cdfb2939435f3b966938cd407b
I20260929 17:53:05.793833 1719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:05.786748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42603 (local address 127.0.98.1:50865)
0929 17:53:05.787113 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:05.787127 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 17:53:05.787261 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:53:05.787476 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:53:05.787478 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:53:05.787680 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:53:05.787800 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:05.787807 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:05.788555 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:05.788558 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:05.789328 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:05.789337 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:05.789853 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:05.789855 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:05.789921 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:05.790404 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:53:05.790417 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:53:05.790778 (+ 361us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:53:05.792350 (+ 1572us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:53:05.792353 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:53:05.792355 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:53:05.792586 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:53:05.792846 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:53:05.792848 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:53:05.792850 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:53:05.792935 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:53:05.793361 (+ 426us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:53:05.793364 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:53:05.793456 (+ 92us) client_negotiation.cc:769] Sending connection context
0929 17:53:05.793553 (+ 97us) client_negotiation.cc:240] Negotiation successful
0929 17:53:05.793671 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":94,"threads_started":1}
I20260929 17:53:05.793906 1559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:05.786843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50865 (local address 127.0.98.62:42603)
0929 17:53:05.786961 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:05.786965 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:05.787196 (+ 231us) server_negotiation.cc:407] Connection header received
0929 17:53:05.787345 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:05.787347 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:05.787385 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:05.787444 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:53:05.787891 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:05.788443 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:05.789437 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:05.789647 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:05.789749 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:05.790893 (+ 1144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:53:05.790897 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:53:05.790899 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:53:05.790920 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:53:05.792254 (+ 1334us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:53:05.792675 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:53:05.792678 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:53:05.792679 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:53:05.792750 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:53:05.793070 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:53:05.793074 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:53:05.793076 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:53:05.793262 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:53:05.793360 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0929 17:53:05.793636 (+ 276us) server_negotiation.cc:299] Negotiation successful
0929 17:53:05.793748 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704375975685-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704375975685-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260929 17:53:05.814093 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskup3Soe/build/debug/bin/kudu
/tmp/dist-test-taskup3Soe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704375975685-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704375975685-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704375975685-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704375975685-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42603
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704375975685-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704375975685-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704375975685-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704375975685-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:53:05.921747 1722 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:53:05.921990 1722 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:53:05.922034 1722 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:53:05.925905 1722 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:53:05.925957 1722 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:53:05.925994 1722 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 17:53:05.926033 1722 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 17:53:05.931237 1722 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704375975685-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704375975685-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704375975685-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704375975685-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:42603
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1722
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704375975685-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:53:05.932230 1722 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:53:05.933190 1722 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:53:05.942168 1727 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:53:05.942138 1728 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:53:05.942157 1730 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:53:05.942404 1722 server_base.cc:1034] running on GCE node
I20260929 17:53:05.942935 1722 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:53:05.943573 1722 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:53:05.944752 1722 hybrid_clock.cc:648] HybridClock initialized: now 1790704385944734 us; error 46 us; skew 500 ppm
Sep 29 17:53:05 dist-test-slave-53nh krb5kdc[1465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704385, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:53:05.947440 1722 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260929 17:53:05.948256 1722 webserver.cc:460] Webserver started at http://127.0.98.2:34303/ using document root <none> and password file <none>
I20260929 17:53:05.948786 1722 fs_manager.cc:359] Metadata directory not provided
I20260929 17:53:05.948843 1722 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:53:05.949047 1722 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:53:05.951016 1722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704375975685-392-0/minicluster-data/ts-1/data/instance:
uuid: "1914a6c9793a4ceca993848ecd50c415"
format_stamp: "Formatted at 2026-09-29 17:53:05 on dist-test-slave-53nh"
server_key: "92ea3e0bafaf0bc3655b74e6bc9133fc"
server_key_iv: "0ec4a521d0917d30d4870103c153d81f"
server_key_version: "encryptionkey@0"
I20260929 17:53:05.951529 1722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704375975685-392-0/minicluster-data/ts-1/wal/instance:
uuid: "1914a6c9793a4ceca993848ecd50c415"
format_stamp: "Formatted at 2026-09-29 17:53:05 on dist-test-slave-53nh"
server_key: "92ea3e0bafaf0bc3655b74e6bc9133fc"
server_key_iv: "0ec4a521d0917d30d4870103c153d81f"
server_key_version: "encryptionkey@0"
I20260929 17:53:05.955051 1722 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260929 17:53:05.957441 1737 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:53:05.958573 1722 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 17:53:05.958673 1722 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704375975685-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704375975685-392-0/minicluster-data/ts-1/wal
uuid: "1914a6c9793a4ceca993848ecd50c415"
format_stamp: "Formatted at 2026-09-29 17:53:05 on dist-test-slave-53nh"
server_key: "92ea3e0bafaf0bc3655b74e6bc9133fc"
server_key_iv: "0ec4a521d0917d30d4870103c153d81f"
server_key_version: "encryptionkey@0"
I20260929 17:53:05.958765 1722 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704375975685-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704375975685-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704375975685-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:53:05.982209 1722 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:53:05.985100 1722 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:53:05.985291 1722 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:53:05.986763 1722 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:53:05.986811 1722 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:53:05.986848 1722 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:53:05.986863 1722 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 17:53:05 dist-test-slave-53nh krb5kdc[1465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704385, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 17:53:05.996706 1722 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42193
I20260929 17:53:05.996778 1852 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42193 every 8 connection(s)
I20260929 17:53:05.997799 1722 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704375975685-392-0/minicluster-data/ts-1/data/info.pb
I20260929 17:53:05.999783 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskup3Soe/build/debug/bin/kudu as pid 1722
I20260929 17:53:05.999874 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704375975685-392-0/minicluster-data/ts-1/wal/instance
I20260929 17:53:05.999982 1559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:05.987473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54289 (local address 127.0.98.62:42603)
0929 17:53:05.987592 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:05.987596 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:05.988638 (+ 1042us) server_negotiation.cc:407] Connection header received
0929 17:53:05.989654 (+ 1016us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:05.989657 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:05.989714 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:05.989805 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:53:05.991367 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:05.992174 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:05.993467 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:05.993741 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:05.993836 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:05.996406 (+ 2570us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:53:05.996410 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:53:05.996413 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:53:05.996443 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:53:05.997978 (+ 1535us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:53:05.998523 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:53:05.998527 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:53:05.998529 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:53:05.998626 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:53:05.998935 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:53:05.998939 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:53:05.998940 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:53:05.999113 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:53:05.999198 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0929 17:53:05.999664 (+ 466us) server_negotiation.cc:299] Negotiation successful
0929 17:53:05.999827 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 17:53:06.000139 392 external_mini_cluster.cc:1428] Setting key b8c01421858521e94f715ecc96bb19d6
I20260929 17:53:06.000766 1746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:05.988032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42603 (local address 127.0.98.2:54289)
0929 17:53:05.988493 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:05.988536 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 17:53:05.989411 (+ 875us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:53:05.990234 (+ 823us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:53:05.990254 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:53:05.990629 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:53:05.991231 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:05.991246 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:05.992301 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:05.992305 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:05.993354 (+ 1049us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:05.993363 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:05.993890 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:05.993894 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:05.994060 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:05.994572 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:53:05.994587 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:53:05.996223 (+ 1636us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:53:05.998078 (+ 1855us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:53:05.998082 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:53:05.998093 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:53:05.998407 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:53:05.998710 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:53:05.998713 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:53:05.998716 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:53:05.998849 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:53:05.999208 (+ 359us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:53:05.999215 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:53:05.999446 (+ 231us) client_negotiation.cc:769] Sending connection context
0929 17:53:05.999622 (+ 176us) client_negotiation.cc:240] Negotiation successful
0929 17:53:05.999923 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":130,"threads_started":1}
I20260929 17:53:06.001935 1853 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42603
I20260929 17:53:06.002068 1853 heartbeater.cc:464] Registering TS with master...
I20260929 17:53:06.002578 1853 heartbeater.cc:511] Master 127.0.98.62:42603 requested a full tablet report, sending...
I20260929 17:53:06.003518 1510 ts_manager.cc:194] Registered new tserver with Master: 1914a6c9793a4ceca993848ecd50c415 (127.0.98.2:42193)
I20260929 17:53:06.004223 1510 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:54289
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260929 17:53:06.010702 1559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:06.003954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51721 (local address 127.0.98.62:42603)
0929 17:53:06.004056 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:06.004061 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:06.004249 (+ 188us) server_negotiation.cc:407] Connection header received
0929 17:53:06.004346 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:06.004350 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:06.004392 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:06.004462 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:53:06.004974 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:06.005711 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:06.006647 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:06.006874 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:06.006955 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:06.007933 (+ 978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:53:06.007936 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:53:06.007939 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:53:06.007962 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:53:06.009303 (+ 1341us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:53:06.009654 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:53:06.009658 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:53:06.009660 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:53:06.009727 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:53:06.009959 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:53:06.009962 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:53:06.009964 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:53:06.010151 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:53:06.010242 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0929 17:53:06.010465 (+ 223us) server_negotiation.cc:299] Negotiation successful
0929 17:53:06.010562 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260929 17:53:06.010721 1860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:06.003884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42603 (local address 127.0.98.2:51721)
0929 17:53:06.004164 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:06.004177 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 17:53:06.004268 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:53:06.004564 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:53:06.004567 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:53:06.004733 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:53:06.004851 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:06.004858 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:06.005872 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:06.005875 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:06.006541 (+ 666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:06.006549 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:06.006966 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:06.006970 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:06.007054 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:06.007496 (+ 442us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:53:06.007509 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:53:06.007806 (+ 297us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:53:06.009396 (+ 1590us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:53:06.009399 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:53:06.009401 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:53:06.009573 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:53:06.009803 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:53:06.009806 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:53:06.009808 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:53:06.009883 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:53:06.010236 (+ 353us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:53:06.010239 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:53:06.010323 (+ 84us) client_negotiation.cc:769] Sending connection context
0929 17:53:06.010405 (+ 82us) client_negotiation.cc:240] Negotiation successful
0929 17:53:06.010529 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":90,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704375975685-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704375975685-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260929 17:53:06.025890 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskup3Soe/build/debug/bin/kudu
/tmp/dist-test-taskup3Soe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704375975685-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704375975685-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704375975685-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704375975685-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42603
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704375975685-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704375975685-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704375975685-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704375975685-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:53:06.132653 1862 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:53:06.132918 1862 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:53:06.132961 1862 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:53:06.136835 1862 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:53:06.136888 1862 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:53:06.136924 1862 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 17:53:06.136960 1862 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 17:53:06.141978 1862 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704375975685-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704375975685-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704375975685-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704375975685-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:42603
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1862
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704375975685-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:53:06.142925 1862 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:53:06.144008 1862 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:53:06.152935 1868 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:53:06.152943 1870 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:53:06.153117 1862 server_base.cc:1034] running on GCE node
W20260929 17:53:06.152946 1867 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:53:06.153514 1862 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:53:06.154089 1862 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:53:06.155261 1862 hybrid_clock.cc:648] HybridClock initialized: now 1790704386155241 us; error 28 us; skew 500 ppm
Sep 29 17:53:06 dist-test-slave-53nh krb5kdc[1465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704386, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:53:06.170291 1862 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260929 17:53:06.171264 1862 webserver.cc:460] Webserver started at http://127.0.98.3:38903/ using document root <none> and password file <none>
I20260929 17:53:06.171831 1862 fs_manager.cc:359] Metadata directory not provided
I20260929 17:53:06.171898 1862 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:53:06.172096 1862 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:53:06.174181 1862 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704375975685-392-0/minicluster-data/ts-2/data/instance:
uuid: "7415a5be86774cc59fd6f269433d6a2b"
format_stamp: "Formatted at 2026-09-29 17:53:06 on dist-test-slave-53nh"
server_key: "66bc17dc2c71417ce1b0a523a65ba02a"
server_key_iv: "51e9f8b30085f0b08e2a1b5ec8f5a0b9"
server_key_version: "encryptionkey@0"
I20260929 17:53:06.174726 1862 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704375975685-392-0/minicluster-data/ts-2/wal/instance:
uuid: "7415a5be86774cc59fd6f269433d6a2b"
format_stamp: "Formatted at 2026-09-29 17:53:06 on dist-test-slave-53nh"
server_key: "66bc17dc2c71417ce1b0a523a65ba02a"
server_key_iv: "51e9f8b30085f0b08e2a1b5ec8f5a0b9"
server_key_version: "encryptionkey@0"
I20260929 17:53:06.178406 1862 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 17:53:06.180970 1877 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:53:06.182155 1862 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 17:53:06.182246 1862 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704375975685-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704375975685-392-0/minicluster-data/ts-2/wal
uuid: "7415a5be86774cc59fd6f269433d6a2b"
format_stamp: "Formatted at 2026-09-29 17:53:06 on dist-test-slave-53nh"
server_key: "66bc17dc2c71417ce1b0a523a65ba02a"
server_key_iv: "51e9f8b30085f0b08e2a1b5ec8f5a0b9"
server_key_version: "encryptionkey@0"
I20260929 17:53:06.182367 1862 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704375975685-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704375975685-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704375975685-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:53:06.192822 1862 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:53:06.195338 1862 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:53:06.195488 1862 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:53:06.196770 1862 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:53:06.196816 1862 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:53:06.196856 1862 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:53:06.196883 1862 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 17:53:06 dist-test-slave-53nh krb5kdc[1465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704386, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 17:53:06.206629 1862 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38471
I20260929 17:53:06.206668 1992 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38471 every 8 connection(s)
I20260929 17:53:06.207531 1862 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704375975685-392-0/minicluster-data/ts-2/data/info.pb
I20260929 17:53:06.209633 1559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:06.197496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40181 (local address 127.0.98.62:42603)
0929 17:53:06.197648 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:06.197654 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:06.198413 (+ 759us) server_negotiation.cc:407] Connection header received
0929 17:53:06.199349 (+ 936us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:06.199353 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:06.199406 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:06.199462 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:53:06.200671 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:06.201437 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:06.203090 (+ 1653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:06.203376 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:06.203456 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:06.206282 (+ 2826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:53:06.206286 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:53:06.206289 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:53:06.206318 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:53:06.207697 (+ 1379us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:53:06.208244 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:53:06.208247 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:53:06.208249 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:53:06.208331 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:53:06.208619 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:53:06.208621 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:53:06.208622 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:53:06.208805 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:53:06.208880 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0929 17:53:06.209419 (+ 539us) server_negotiation.cc:299] Negotiation successful
0929 17:53:06.209516 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 17:53:06.210389 1886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:06.197842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42603 (local address 127.0.98.3:40181)
0929 17:53:06.198258 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:06.198305 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 17:53:06.199127 (+ 822us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:53:06.199779 (+ 652us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:53:06.199785 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:53:06.200084 (+ 299us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:53:06.200514 (+ 430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:06.200527 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:06.201549 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:06.201552 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:06.202971 (+ 1419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:06.202983 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:06.203514 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:06.203518 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:06.203717 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:06.204278 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:53:06.204300 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:53:06.206098 (+ 1798us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:53:06.207799 (+ 1701us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:53:06.207804 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:53:06.207819 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:53:06.208128 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:53:06.208411 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:53:06.208414 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:53:06.208417 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:53:06.208549 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:53:06.208916 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:53:06.208921 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:53:06.209211 (+ 290us) client_negotiation.cc:769] Sending connection context
0929 17:53:06.209380 (+ 169us) client_negotiation.cc:240] Negotiation successful
0929 17:53:06.209644 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":100,"threads_started":1}
I20260929 17:53:06.211450 1993 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42603
I20260929 17:53:06.211522 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskup3Soe/build/debug/bin/kudu as pid 1862
I20260929 17:53:06.211580 1993 heartbeater.cc:464] Registering TS with master...
I20260929 17:53:06.211621 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704375975685-392-0/minicluster-data/ts-2/wal/instance
I20260929 17:53:06.211908 392 external_mini_cluster.cc:1428] Setting key 4c963df6065b6b56cb9a8f098c718a00
I20260929 17:53:06.212047 1993 heartbeater.cc:511] Master 127.0.98.62:42603 requested a full tablet report, sending...
I20260929 17:53:06.212972 1509 ts_manager.cc:194] Registered new tserver with Master: 7415a5be86774cc59fd6f269433d6a2b (127.0.98.3:38471)
I20260929 17:53:06.213634 1509 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:40181
I20260929 17:53:06.214131 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 17:53:06.220498 1999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:06.213683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42603 (local address 127.0.98.3:41355)
0929 17:53:06.213985 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:06.213999 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 17:53:06.214067 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:53:06.214252 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:53:06.214254 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:53:06.214442 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:53:06.214548 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:06.214555 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:06.215276 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:06.215280 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:06.216067 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:06.216076 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:06.216614 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:06.216617 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:06.216704 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:06.217221 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:53:06.217235 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:53:06.217515 (+ 280us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:53:06.219153 (+ 1638us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:53:06.219157 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:53:06.219159 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:53:06.219382 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:53:06.219622 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:53:06.219625 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:53:06.219627 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:53:06.219712 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:53:06.220013 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:53:06.220015 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:53:06.220101 (+ 86us) client_negotiation.cc:769] Sending connection context
0929 17:53:06.220199 (+ 98us) client_negotiation.cc:240] Negotiation successful
0929 17:53:06.220331 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":98,"threads_started":1}
I20260929 17:53:06.220595 1559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:06.213928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41355 (local address 127.0.98.62:42603)
0929 17:53:06.214085 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:06.214088 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:06.214100 (+ 12us) server_negotiation.cc:407] Connection header received
0929 17:53:06.214139 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:06.214140 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:06.214172 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:06.214236 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:53:06.214643 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:06.215159 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:06.216193 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:06.216501 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:06.216599 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:06.217636 (+ 1037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:53:06.217640 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:53:06.217642 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:53:06.217666 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:53:06.219049 (+ 1383us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:53:06.219466 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:53:06.219469 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:53:06.219470 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:53:06.219540 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:53:06.219800 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:53:06.219802 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:53:06.219804 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:53:06.219946 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:53:06.220013 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0929 17:53:06.220334 (+ 321us) server_negotiation.cc:299] Negotiation successful
0929 17:53:06.220450 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
Sep 29 17:53:06 dist-test-slave-53nh krb5kdc[1465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704386, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 17:53:06 dist-test-slave-53nh krb5kdc[1465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704386, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 17:53:06.238759 1559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:06.231454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36876 (local address 127.0.98.62:42603)
0929 17:53:06.231591 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:06.231594 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:06.231606 (+ 12us) server_negotiation.cc:407] Connection header received
0929 17:53:06.231643 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:06.231646 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:06.231684 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:06.231751 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:53:06.232450 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:06.232958 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:06.233817 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:06.234018 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:06.234085 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:06.236220 (+ 2135us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:53:06.236223 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:53:06.236225 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:53:06.236249 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:53:06.237425 (+ 1176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:53:06.237817 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:53:06.237819 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:53:06.237821 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:53:06.237892 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:53:06.238101 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:53:06.238103 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:53:06.238104 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:53:06.238274 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:53:06.238368 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0929 17:53:06.238547 (+ 179us) server_negotiation.cc:299] Negotiation successful
0929 17:53:06.238636 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 17:53:06.241654 1510 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36876:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 17:53:06.244772 1510 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 17:53:06.257903 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:06.252632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42193 (local address 127.0.0.1:54864)
0929 17:53:06.252932 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:06.252975 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 17:53:06.253136 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:53:06.253674 (+ 538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:53:06.253681 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:53:06.253705 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:53:06.253856 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:06.253864 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:06.255142 (+ 1278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:06.255145 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:06.256240 (+ 1095us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:06.256247 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:06.257335 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:06.257339 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:06.257464 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:06.257508 (+ 44us) client_negotiation.cc:769] Sending connection context
0929 17:53:06.257610 (+ 102us) client_negotiation.cc:240] Negotiation successful
0929 17:53:06.257738 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"mutex_wait_us":4,"thread_start_us":69,"threads_started":1}
I20260929 17:53:06.258320 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:06.252837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54864 (local address 127.0.98.2:42193)
0929 17:53:06.253187 (+ 350us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:06.253198 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:06.253221 (+ 23us) server_negotiation.cc:407] Connection header received
0929 17:53:06.253299 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:06.253304 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:06.253513 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:06.253669 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:53:06.253963 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:06.255030 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:06.256362 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:06.257233 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:06.257340 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:06.257509 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0929 17:53:06.258083 (+ 574us) server_negotiation.cc:299] Negotiation successful
0929 17:53:06.258186 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":254,"thread_start_us":109,"threads_started":1}
I20260929 17:53:06.259408 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:06.253378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40623 (local address 127.0.0.1:35466)
0929 17:53:06.253790 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:06.253802 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 17:53:06.253873 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:53:06.255278 (+ 1405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:53:06.255282 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:53:06.255301 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:53:06.255410 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:06.255417 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:06.256917 (+ 1500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:06.256920 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:06.258076 (+ 1156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:06.258089 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:06.259034 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:06.259037 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:06.259119 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:06.259142 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 17:53:06.259203 (+ 61us) client_negotiation.cc:240] Negotiation successful
0929 17:53:06.259276 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":344,"mutex_wait_us":19,"thread_start_us":93,"threads_started":1}
I20260929 17:53:06.259441 2015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:06.254192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35466 (local address 127.0.98.1:40623)
0929 17:53:06.254858 (+ 666us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:06.254871 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:06.254892 (+ 21us) server_negotiation.cc:407] Connection header received
0929 17:53:06.254956 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:06.254960 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:06.255157 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:06.255278 (+ 121us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:53:06.255533 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:06.256797 (+ 1264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:06.258219 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:06.258915 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:06.259019 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:06.259169 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0929 17:53:06.259227 (+ 58us) server_negotiation.cc:299] Negotiation successful
0929 17:53:06.259325 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":573,"thread_start_us":95,"threads_started":1}
I20260929 17:53:06.261410 1787 tablet_service.cc:1467] Processing CreateTablet for tablet dee751ad1aca44f8b8089e50fe97e7ee (DEFAULT_TABLE table=test-table [id=b396f96fcf894ffea40efa8e51b2a2f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 17:53:06.261880 1647 tablet_service.cc:1467] Processing CreateTablet for tablet dee751ad1aca44f8b8089e50fe97e7ee (DEFAULT_TABLE table=test-table [id=b396f96fcf894ffea40efa8e51b2a2f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 17:53:06.262416 1787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dee751ad1aca44f8b8089e50fe97e7ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:53:06.262755 1647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dee751ad1aca44f8b8089e50fe97e7ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:53:06.268782 2016 tablet_bootstrap.cc:492] T dee751ad1aca44f8b8089e50fe97e7ee P 1914a6c9793a4ceca993848ecd50c415: Bootstrap starting.
I20260929 17:53:06.270021 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:06.253666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38471 (local address 127.0.0.1:52134)
0929 17:53:06.254538 (+ 872us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:06.254551 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 17:53:06.254629 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:53:06.255921 (+ 1292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:53:06.255924 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:53:06.255941 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:53:06.256015 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:06.256021 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:06.257281 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:06.257284 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:06.258319 (+ 1035us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:06.258329 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:06.269537 (+ 11208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:06.269541 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:06.269664 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:06.269690 (+ 26us) client_negotiation.cc:769] Sending connection context
0929 17:53:06.269779 (+ 89us) client_negotiation.cc:240] Negotiation successful
0929 17:53:06.269874 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":814,"mutex_wait_us":1,"thread_start_us":88,"threads_started":1}
I20260929 17:53:06.270464 2017 tablet_bootstrap.cc:492] T dee751ad1aca44f8b8089e50fe97e7ee P 0ff660fa6d574e1a83e0ccc53672f485: Bootstrap starting.
I20260929 17:53:06.270529 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:06.254164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52134 (local address 127.0.98.3:38471)
0929 17:53:06.255517 (+ 1353us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:06.255527 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:06.255546 (+ 19us) server_negotiation.cc:407] Connection header received
0929 17:53:06.255613 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:06.255617 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:06.255792 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:06.255929 (+ 137us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:53:06.256115 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:06.257179 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:06.258673 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:06.269411 (+ 10738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:06.269525 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:06.269701 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0929 17:53:06.270192 (+ 491us) server_negotiation.cc:299] Negotiation successful
0929 17:53:06.270338 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1254,"thread_start_us":116,"threads_started":1}
I20260929 17:53:06.271396 2016 tablet_bootstrap.cc:654] T dee751ad1aca44f8b8089e50fe97e7ee P 1914a6c9793a4ceca993848ecd50c415: Neither blocks nor log segments found. Creating new log.
I20260929 17:53:06.272097 2016 log.cc:824] T dee751ad1aca44f8b8089e50fe97e7ee P 1914a6c9793a4ceca993848ecd50c415: Log is configured to *not* fsync() on all Append() calls
I20260929 17:53:06.272425 2017 tablet_bootstrap.cc:654] T dee751ad1aca44f8b8089e50fe97e7ee P 0ff660fa6d574e1a83e0ccc53672f485: Neither blocks nor log segments found. Creating new log.
I20260929 17:53:06.273221 2017 log.cc:824] T dee751ad1aca44f8b8089e50fe97e7ee P 0ff660fa6d574e1a83e0ccc53672f485: Log is configured to *not* fsync() on all Append() calls
I20260929 17:53:06.273566 1927 tablet_service.cc:1467] Processing CreateTablet for tablet dee751ad1aca44f8b8089e50fe97e7ee (DEFAULT_TABLE table=test-table [id=b396f96fcf894ffea40efa8e51b2a2f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 17:53:06.274037 2016 tablet_bootstrap.cc:492] T dee751ad1aca44f8b8089e50fe97e7ee P 1914a6c9793a4ceca993848ecd50c415: No bootstrap required, opened a new log
I20260929 17:53:06.274217 2016 ts_tablet_manager.cc:1397] T dee751ad1aca44f8b8089e50fe97e7ee P 1914a6c9793a4ceca993848ecd50c415: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 17:53:06.274530 1927 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dee751ad1aca44f8b8089e50fe97e7ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:53:06.275102 2017 tablet_bootstrap.cc:492] T dee751ad1aca44f8b8089e50fe97e7ee P 0ff660fa6d574e1a83e0ccc53672f485: No bootstrap required, opened a new log
I20260929 17:53:06.275279 2017 ts_tablet_manager.cc:1397] T dee751ad1aca44f8b8089e50fe97e7ee P 0ff660fa6d574e1a83e0ccc53672f485: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260929 17:53:06.277379 2016 raft_consensus.cc:348] T dee751ad1aca44f8b8089e50fe97e7ee P 1914a6c9793a4ceca993848ecd50c415 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1914a6c9793a4ceca993848ecd50c415" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42193 } } peers { permanent_uuid: "0ff660fa6d574e1a83e0ccc53672f485" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40623 } } peers { permanent_uuid: "7415a5be86774cc59fd6f269433d6a2b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38471 } }
I20260929 17:53:06.277587 2016 raft_consensus.cc:374] T dee751ad1aca44f8b8089e50fe97e7ee P 1914a6c9793a4ceca993848ecd50c415 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:53:06.277643 2016 raft_consensus.cc:729] T dee751ad1aca44f8b8089e50fe97e7ee P 1914a6c9793a4ceca993848ecd50c415 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1914a6c9793a4ceca993848ecd50c415, State: Initialized, Role: FOLLOWER
I20260929 17:53:06.278049 2016 consensus_queue.cc:260] T dee751ad1aca44f8b8089e50fe97e7ee P 1914a6c9793a4ceca993848ecd50c415 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1914a6c9793a4ceca993848ecd50c415" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42193 } } peers { permanent_uuid: "0ff660fa6d574e1a83e0ccc53672f485" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40623 } } peers { permanent_uuid: "7415a5be86774cc59fd6f269433d6a2b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38471 } }
I20260929 17:53:06.278674 1853 heartbeater.cc:503] Master 127.0.98.62:42603 was elected leader, sending a full tablet report...
I20260929 17:53:06.279016 2016 ts_tablet_manager.cc:1428] T dee751ad1aca44f8b8089e50fe97e7ee P 1914a6c9793a4ceca993848ecd50c415: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260929 17:53:06.278987 2017 raft_consensus.cc:348] T dee751ad1aca44f8b8089e50fe97e7ee P 0ff660fa6d574e1a83e0ccc53672f485 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1914a6c9793a4ceca993848ecd50c415" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42193 } } peers { permanent_uuid: "0ff660fa6d574e1a83e0ccc53672f485" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40623 } } peers { permanent_uuid: "7415a5be86774cc59fd6f269433d6a2b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38471 } }
I20260929 17:53:06.279223 2017 raft_consensus.cc:374] T dee751ad1aca44f8b8089e50fe97e7ee P 0ff660fa6d574e1a83e0ccc53672f485 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:53:06.279279 2017 raft_consensus.cc:729] T dee751ad1aca44f8b8089e50fe97e7ee P 0ff660fa6d574e1a83e0ccc53672f485 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ff660fa6d574e1a83e0ccc53672f485, State: Initialized, Role: FOLLOWER
I20260929 17:53:06.279699 2017 consensus_queue.cc:260] T dee751ad1aca44f8b8089e50fe97e7ee P 0ff660fa6d574e1a83e0ccc53672f485 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1914a6c9793a4ceca993848ecd50c415" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42193 } } peers { permanent_uuid: "0ff660fa6d574e1a83e0ccc53672f485" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40623 } } peers { permanent_uuid: "7415a5be86774cc59fd6f269433d6a2b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38471 } }
I20260929 17:53:06.280308 1713 heartbeater.cc:503] Master 127.0.98.62:42603 was elected leader, sending a full tablet report...
I20260929 17:53:06.280628 2017 ts_tablet_manager.cc:1428] T dee751ad1aca44f8b8089e50fe97e7ee P 0ff660fa6d574e1a83e0ccc53672f485: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260929 17:53:06.281909 2022 tablet_bootstrap.cc:492] T dee751ad1aca44f8b8089e50fe97e7ee P 7415a5be86774cc59fd6f269433d6a2b: Bootstrap starting.
W20260929 17:53:06.282178 1714 tablet.cc:2369] T dee751ad1aca44f8b8089e50fe97e7ee P 0ff660fa6d574e1a83e0ccc53672f485: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 17:53:06.284046 2022 tablet_bootstrap.cc:654] T dee751ad1aca44f8b8089e50fe97e7ee P 7415a5be86774cc59fd6f269433d6a2b: Neither blocks nor log segments found. Creating new log.
I20260929 17:53:06.284770 2022 log.cc:824] T dee751ad1aca44f8b8089e50fe97e7ee P 7415a5be86774cc59fd6f269433d6a2b: Log is configured to *not* fsync() on all Append() calls
I20260929 17:53:06.286489 2022 tablet_bootstrap.cc:492] T dee751ad1aca44f8b8089e50fe97e7ee P 7415a5be86774cc59fd6f269433d6a2b: No bootstrap required, opened a new log
I20260929 17:53:06.286657 2022 ts_tablet_manager.cc:1397] T dee751ad1aca44f8b8089e50fe97e7ee P 7415a5be86774cc59fd6f269433d6a2b: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260929 17:53:06.289453 2022 raft_consensus.cc:348] T dee751ad1aca44f8b8089e50fe97e7ee P 7415a5be86774cc59fd6f269433d6a2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1914a6c9793a4ceca993848ecd50c415" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42193 } } peers { permanent_uuid: "0ff660fa6d574e1a83e0ccc53672f485" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40623 } } peers { permanent_uuid: "7415a5be86774cc59fd6f269433d6a2b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38471 } }
I20260929 17:53:06.289659 2022 raft_consensus.cc:374] T dee751ad1aca44f8b8089e50fe97e7ee P 7415a5be86774cc59fd6f269433d6a2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:53:06.289710 2022 raft_consensus.cc:729] T dee751ad1aca44f8b8089e50fe97e7ee P 7415a5be86774cc59fd6f269433d6a2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7415a5be86774cc59fd6f269433d6a2b, State: Initialized, Role: FOLLOWER
I20260929 17:53:06.290055 2022 consensus_queue.cc:260] T dee751ad1aca44f8b8089e50fe97e7ee P 7415a5be86774cc59fd6f269433d6a2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1914a6c9793a4ceca993848ecd50c415" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42193 } } peers { permanent_uuid: "0ff660fa6d574e1a83e0ccc53672f485" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40623 } } peers { permanent_uuid: "7415a5be86774cc59fd6f269433d6a2b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38471 } }
I20260929 17:53:06.290592 1993 heartbeater.cc:503] Master 127.0.98.62:42603 was elected leader, sending a full tablet report...
I20260929 17:53:06.290938 2022 ts_tablet_manager.cc:1428] T dee751ad1aca44f8b8089e50fe97e7ee P 7415a5be86774cc59fd6f269433d6a2b: Time spent starting tablet: real 0.004s user 0.004s sys 0.001s
I20260929 17:53:06.374732 2020 raft_consensus.cc:482] T dee751ad1aca44f8b8089e50fe97e7ee P 1914a6c9793a4ceca993848ecd50c415 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 17:53:06.374915 2020 raft_consensus.cc:504] T dee751ad1aca44f8b8089e50fe97e7ee P 1914a6c9793a4ceca993848ecd50c415 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1914a6c9793a4ceca993848ecd50c415" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42193 } } peers { permanent_uuid: "0ff660fa6d574e1a83e0ccc53672f485" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40623 } } peers { permanent_uuid: "7415a5be86774cc59fd6f269433d6a2b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38471 } }
I20260929 17:53:06.375962 2020 leader_election.cc:288] T dee751ad1aca44f8b8089e50fe97e7ee P 1914a6c9793a4ceca993848ecd50c415 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0ff660fa6d574e1a83e0ccc53672f485 (127.0.98.1:40623), 7415a5be86774cc59fd6f269433d6a2b (127.0.98.3:38471)
I20260929 17:53:06.380864 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:06.376621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51233 (local address 127.0.98.3:38471)
0929 17:53:06.376751 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:06.376755 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:06.376929 (+ 174us) server_negotiation.cc:407] Connection header received
0929 17:53:06.377041 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:06.377044 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:06.377090 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:06.377167 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:53:06.377410 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:06.378035 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:06.379679 (+ 1644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:06.380277 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:06.380381 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:06.380444 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0929 17:53:06.380668 (+ 224us) server_negotiation.cc:299] Negotiation successful
0929 17:53:06.380757 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 17:53:06.380864 2015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:06.376309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50861 (local address 127.0.98.1:40623)
0929 17:53:06.376435 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:06.376439 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:06.376454 (+ 15us) server_negotiation.cc:407] Connection header received
0929 17:53:06.376505 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:06.376508 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:06.376555 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:06.376625 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:53:06.376922 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:06.377673 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:06.379679 (+ 2006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:06.380280 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:06.380381 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:06.380444 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0929 17:53:06.380668 (+ 224us) server_negotiation.cc:299] Negotiation successful
0929 17:53:06.380736 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 17:53:06.380903 1746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:06.376199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40623 (local address 127.0.98.2:50861)
0929 17:53:06.376322 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:06.376336 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 17:53:06.376428 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:53:06.376656 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:53:06.376659 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:53:06.376682 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:53:06.376807 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:06.376815 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:06.377762 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:06.377765 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:06.379557 (+ 1792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:06.379567 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:06.380388 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:06.380392 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:06.380540 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:06.380581 (+ 41us) client_negotiation.cc:769] Sending connection context
0929 17:53:06.380648 (+ 67us) client_negotiation.cc:240] Negotiation successful
0929 17:53:06.380730 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42,"mutex_wait_us":150}
I20260929 17:53:06.380919 2025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:06.376470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38471 (local address 127.0.98.2:51233)
0929 17:53:06.376854 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:06.376867 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 17:53:06.376947 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:53:06.377183 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:53:06.377185 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:53:06.377198 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:53:06.377297 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:06.377301 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:06.378137 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:06.378140 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:06.379563 (+ 1423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:06.379569 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:06.380388 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:06.380392 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:06.380535 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:06.380574 (+ 39us) client_negotiation.cc:769] Sending connection context
0929 17:53:06.380648 (+ 74us) client_negotiation.cc:240] Negotiation successful
0929 17:53:06.380730 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":326,"mutex_wait_us":93,"thread_start_us":101,"threads_started":1}
I20260929 17:53:06.381570 1947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dee751ad1aca44f8b8089e50fe97e7ee" candidate_uuid: "1914a6c9793a4ceca993848ecd50c415" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7415a5be86774cc59fd6f269433d6a2b" is_pre_election: true
I20260929 17:53:06.381633 1667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dee751ad1aca44f8b8089e50fe97e7ee" candidate_uuid: "1914a6c9793a4ceca993848ecd50c415" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ff660fa6d574e1a83e0ccc53672f485" is_pre_election: true
I20260929 17:53:06.381875 1947 raft_consensus.cc:2454] T dee751ad1aca44f8b8089e50fe97e7ee P 7415a5be86774cc59fd6f269433d6a2b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1914a6c9793a4ceca993848ecd50c415 in term 0.
I20260929 17:53:06.381887 1667 raft_consensus.cc:2454] T dee751ad1aca44f8b8089e50fe97e7ee P 0ff660fa6d574e1a83e0ccc53672f485 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1914a6c9793a4ceca993848ecd50c415 in term 0.
I20260929 17:53:06.382323 1741 leader_election.cc:302] T dee751ad1aca44f8b8089e50fe97e7ee P 1914a6c9793a4ceca993848ecd50c415 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1914a6c9793a4ceca993848ecd50c415, 7415a5be86774cc59fd6f269433d6a2b; no voters:
I20260929 17:53:06.382555 2020 raft_consensus.cc:2781] T dee751ad1aca44f8b8089e50fe97e7ee P 1914a6c9793a4ceca993848ecd50c415 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 17:53:06.382630 2020 raft_consensus.cc:482] T dee751ad1aca44f8b8089e50fe97e7ee P 1914a6c9793a4ceca993848ecd50c415 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 17:53:06.382675 2020 raft_consensus.cc:3037] T dee751ad1aca44f8b8089e50fe97e7ee P 1914a6c9793a4ceca993848ecd50c415 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:53:06.383694 2020 raft_consensus.cc:504] T dee751ad1aca44f8b8089e50fe97e7ee P 1914a6c9793a4ceca993848ecd50c415 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1914a6c9793a4ceca993848ecd50c415" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42193 } } peers { permanent_uuid: "0ff660fa6d574e1a83e0ccc53672f485" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40623 } } peers { permanent_uuid: "7415a5be86774cc59fd6f269433d6a2b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38471 } }
I20260929 17:53:06.384014 2020 leader_election.cc:288] T dee751ad1aca44f8b8089e50fe97e7ee P 1914a6c9793a4ceca993848ecd50c415 [CANDIDATE]: Term 1 election: Requested vote from peers 0ff660fa6d574e1a83e0ccc53672f485 (127.0.98.1:40623), 7415a5be86774cc59fd6f269433d6a2b (127.0.98.3:38471)
I20260929 17:53:06.384330 1667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dee751ad1aca44f8b8089e50fe97e7ee" candidate_uuid: "1914a6c9793a4ceca993848ecd50c415" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ff660fa6d574e1a83e0ccc53672f485"
I20260929 17:53:06.384378 1947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dee751ad1aca44f8b8089e50fe97e7ee" candidate_uuid: "1914a6c9793a4ceca993848ecd50c415" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7415a5be86774cc59fd6f269433d6a2b"
I20260929 17:53:06.384439 1667 raft_consensus.cc:3037] T dee751ad1aca44f8b8089e50fe97e7ee P 0ff660fa6d574e1a83e0ccc53672f485 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:53:06.384498 1947 raft_consensus.cc:3037] T dee751ad1aca44f8b8089e50fe97e7ee P 7415a5be86774cc59fd6f269433d6a2b [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:53:06.385515 1667 raft_consensus.cc:2454] T dee751ad1aca44f8b8089e50fe97e7ee P 0ff660fa6d574e1a83e0ccc53672f485 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1914a6c9793a4ceca993848ecd50c415 in term 1.
I20260929 17:53:06.385771 1741 leader_election.cc:302] T dee751ad1aca44f8b8089e50fe97e7ee P 1914a6c9793a4ceca993848ecd50c415 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ff660fa6d574e1a83e0ccc53672f485, 1914a6c9793a4ceca993848ecd50c415; no voters:
I20260929 17:53:06.385815 1947 raft_consensus.cc:2454] T dee751ad1aca44f8b8089e50fe97e7ee P 7415a5be86774cc59fd6f269433d6a2b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1914a6c9793a4ceca993848ecd50c415 in term 1.
I20260929 17:53:06.385917 2020 raft_consensus.cc:2781] T dee751ad1aca44f8b8089e50fe97e7ee P 1914a6c9793a4ceca993848ecd50c415 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 17:53:06.386113 2020 raft_consensus.cc:686] T dee751ad1aca44f8b8089e50fe97e7ee P 1914a6c9793a4ceca993848ecd50c415 [term 1 LEADER]: Becoming Leader. State: Replica: 1914a6c9793a4ceca993848ecd50c415, State: Running, Role: LEADER
I20260929 17:53:06.386453 2020 consensus_queue.cc:237] T dee751ad1aca44f8b8089e50fe97e7ee P 1914a6c9793a4ceca993848ecd50c415 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1914a6c9793a4ceca993848ecd50c415" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42193 } } peers { permanent_uuid: "0ff660fa6d574e1a83e0ccc53672f485" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40623 } } peers { permanent_uuid: "7415a5be86774cc59fd6f269433d6a2b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38471 } }
I20260929 17:53:06.389441 1509 catalog_manager.cc:5495] T dee751ad1aca44f8b8089e50fe97e7ee P 1914a6c9793a4ceca993848ecd50c415 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1914a6c9793a4ceca993848ecd50c415 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "1914a6c9793a4ceca993848ecd50c415" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1914a6c9793a4ceca993848ecd50c415" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0ff660fa6d574e1a83e0ccc53672f485" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40623 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7415a5be86774cc59fd6f269433d6a2b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38471 } health_report { overall_health: UNKNOWN } } }
I20260929 17:53:06.414901 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:06.411500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54876 (local address 127.0.98.2:42193)
0929 17:53:06.411626 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:06.411630 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:06.411647 (+ 17us) server_negotiation.cc:407] Connection header received
0929 17:53:06.411838 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:06.411840 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:06.411895 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:06.411985 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 17:53:06.412207 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:06.412734 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:06.413607 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:06.413833 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:06.413902 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:06.414158 (+ 256us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 17:53:06.414565 (+ 407us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 17:53:06.414644 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0929 17:53:06.414712 (+ 68us) server_negotiation.cc:299] Negotiation successful
0929 17:53:06.414773 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260929 17:53:06.421048 1947 raft_consensus.cc:1261] T dee751ad1aca44f8b8089e50fe97e7ee P 7415a5be86774cc59fd6f269433d6a2b [term 1 FOLLOWER]: Refusing update from remote peer 1914a6c9793a4ceca993848ecd50c415: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 17:53:06.421048 1667 raft_consensus.cc:1261] T dee751ad1aca44f8b8089e50fe97e7ee P 0ff660fa6d574e1a83e0ccc53672f485 [term 1 FOLLOWER]: Refusing update from remote peer 1914a6c9793a4ceca993848ecd50c415: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 17:53:06.421722 2026 consensus_queue.cc:1035] T dee751ad1aca44f8b8089e50fe97e7ee P 1914a6c9793a4ceca993848ecd50c415 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ff660fa6d574e1a83e0ccc53672f485" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 17:53:06.421890 2020 consensus_queue.cc:1035] T dee751ad1aca44f8b8089e50fe97e7ee P 1914a6c9793a4ceca993848ecd50c415 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7415a5be86774cc59fd6f269433d6a2b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 17:53:06.428819 2029 mvcc.cc:205] Tried to move back new op lower bound from 7334725166773788672 to 7334725166641385472. Current Snapshot: MvccSnapshot[applied={T|T < 7334725166773788672}]
I20260929 17:53:06.430236 2030 mvcc.cc:205] Tried to move back new op lower bound from 7334725166773788672 to 7334725166641385472. Current Snapshot: MvccSnapshot[applied={T|T < 7334725166773788672}]
I20260929 17:53:06.431536 2031 mvcc.cc:205] Tried to move back new op lower bound from 7334725166773788672 to 7334725166641385472. Current Snapshot: MvccSnapshot[applied={T|T < 7334725166773788672}]
I20260929 17:53:06.441094 1509 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36876:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 17:53:06.441278 1509 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36876:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 17:53:06.443881 1509 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bf0bd8ed30e946f49cc31cdf2f3f5424: Sending DeleteTablet for 3 replicas of tablet dee751ad1aca44f8b8089e50fe97e7ee
I20260929 17:53:06.444571 1787 tablet_service.cc:1514] Processing DeleteTablet for tablet dee751ad1aca44f8b8089e50fe97e7ee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 17:53:06 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54864
I20260929 17:53:06.444658 1647 tablet_service.cc:1514] Processing DeleteTablet for tablet dee751ad1aca44f8b8089e50fe97e7ee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 17:53:06 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35466
I20260929 17:53:06.444779 1927 tablet_service.cc:1514] Processing DeleteTablet for tablet dee751ad1aca44f8b8089e50fe97e7ee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 17:53:06 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52134
I20260929 17:53:06.445295 2038 tablet_replica.cc:330] T dee751ad1aca44f8b8089e50fe97e7ee P 1914a6c9793a4ceca993848ecd50c415: stopping tablet replica
I20260929 17:53:06.445302 2040 tablet_replica.cc:330] T dee751ad1aca44f8b8089e50fe97e7ee P 7415a5be86774cc59fd6f269433d6a2b: stopping tablet replica
I20260929 17:53:06.445523 1509 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36876:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260929 17:53:06.445603 2038 raft_consensus.cc:2229] T dee751ad1aca44f8b8089e50fe97e7ee P 1914a6c9793a4ceca993848ecd50c415 [term 1 LEADER]: Raft consensus shutting down.
W20260929 17:53:06.445887 1509 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 17:53:06.446012 2038 raft_consensus.cc:2258] T dee751ad1aca44f8b8089e50fe97e7ee P 1914a6c9793a4ceca993848ecd50c415 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 17:53:06.446709 2040 raft_consensus.cc:2229] T dee751ad1aca44f8b8089e50fe97e7ee P 7415a5be86774cc59fd6f269433d6a2b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 17:53:06.447014 2040 raft_consensus.cc:2258] T dee751ad1aca44f8b8089e50fe97e7ee P 7415a5be86774cc59fd6f269433d6a2b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 17:53:06.448318 2040 ts_tablet_manager.cc:1905] T dee751ad1aca44f8b8089e50fe97e7ee P 7415a5be86774cc59fd6f269433d6a2b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 17:53:06.450889 1787 tablet_service.cc:1467] Processing CreateTablet for tablet 47dd32a1c7e44fe5882f0dc5aa078ce2 (DEFAULT_TABLE table=test-table [id=8da611cccdd948c2ab776776371ec16f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 17:53:06.450893 1647 tablet_service.cc:1467] Processing CreateTablet for tablet 47dd32a1c7e44fe5882f0dc5aa078ce2 (DEFAULT_TABLE table=test-table [id=8da611cccdd948c2ab776776371ec16f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 17:53:06.451074 1927 tablet_service.cc:1467] Processing CreateTablet for tablet 47dd32a1c7e44fe5882f0dc5aa078ce2 (DEFAULT_TABLE table=test-table [id=8da611cccdd948c2ab776776371ec16f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 17:53:06.451166 1647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47dd32a1c7e44fe5882f0dc5aa078ce2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:53:06.451169 1787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47dd32a1c7e44fe5882f0dc5aa078ce2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:53:06.451292 1927 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47dd32a1c7e44fe5882f0dc5aa078ce2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:53:06.451795 2038 ts_tablet_manager.cc:1905] T dee751ad1aca44f8b8089e50fe97e7ee P 1914a6c9793a4ceca993848ecd50c415: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 17:53:06.453502 2022 tablet_bootstrap.cc:492] T 47dd32a1c7e44fe5882f0dc5aa078ce2 P 7415a5be86774cc59fd6f269433d6a2b: Bootstrap starting.
I20260929 17:53:06.453503 2016 tablet_bootstrap.cc:492] T 47dd32a1c7e44fe5882f0dc5aa078ce2 P 1914a6c9793a4ceca993848ecd50c415: Bootstrap starting.
I20260929 17:53:06.454384 2022 tablet_bootstrap.cc:654] T 47dd32a1c7e44fe5882f0dc5aa078ce2 P 7415a5be86774cc59fd6f269433d6a2b: Neither blocks nor log segments found. Creating new log.
I20260929 17:53:06.454465 2016 tablet_bootstrap.cc:654] T 47dd32a1c7e44fe5882f0dc5aa078ce2 P 1914a6c9793a4ceca993848ecd50c415: Neither blocks nor log segments found. Creating new log.
I20260929 17:53:06.454922 2017 tablet_bootstrap.cc:492] T 47dd32a1c7e44fe5882f0dc5aa078ce2 P 0ff660fa6d574e1a83e0ccc53672f485: Bootstrap starting.
I20260929 17:53:06.455561 2022 tablet_bootstrap.cc:492] T 47dd32a1c7e44fe5882f0dc5aa078ce2 P 7415a5be86774cc59fd6f269433d6a2b: No bootstrap required, opened a new log
I20260929 17:53:06.455662 2022 ts_tablet_manager.cc:1397] T 47dd32a1c7e44fe5882f0dc5aa078ce2 P 7415a5be86774cc59fd6f269433d6a2b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20260929 17:53:06.456113 2022 raft_consensus.cc:348] T 47dd32a1c7e44fe5882f0dc5aa078ce2 P 7415a5be86774cc59fd6f269433d6a2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ff660fa6d574e1a83e0ccc53672f485" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40623 } } peers { permanent_uuid: "1914a6c9793a4ceca993848ecd50c415" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42193 } } peers { permanent_uuid: "7415a5be86774cc59fd6f269433d6a2b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38471 } }
I20260929 17:53:06.456218 2022 raft_consensus.cc:374] T 47dd32a1c7e44fe5882f0dc5aa078ce2 P 7415a5be86774cc59fd6f269433d6a2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:53:06.456248 2022 raft_consensus.cc:729] T 47dd32a1c7e44fe5882f0dc5aa078ce2 P 7415a5be86774cc59fd6f269433d6a2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7415a5be86774cc59fd6f269433d6a2b, State: Initialized, Role: FOLLOWER
I20260929 17:53:06.456315 2017 tablet_bootstrap.cc:654] T 47dd32a1c7e44fe5882f0dc5aa078ce2 P 0ff660fa6d574e1a83e0ccc53672f485: Neither blocks nor log segments found. Creating new log.
I20260929 17:53:06.456378 2022 consensus_queue.cc:260] T 47dd32a1c7e44fe5882f0dc5aa078ce2 P 7415a5be86774cc59fd6f269433d6a2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ff660fa6d574e1a83e0ccc53672f485" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40623 } } peers { permanent_uuid: "1914a6c9793a4ceca993848ecd50c415" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42193 } } peers { permanent_uuid: "7415a5be86774cc59fd6f269433d6a2b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38471 } }
I20260929 17:53:06.456624 2022 ts_tablet_manager.cc:1428] T 47dd32a1c7e44fe5882f0dc5aa078ce2 P 7415a5be86774cc59fd6f269433d6a2b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260929 17:53:06.456965 2040 ts_tablet_manager.cc:1918] T dee751ad1aca44f8b8089e50fe97e7ee P 7415a5be86774cc59fd6f269433d6a2b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 17:53:06.457082 2040 log.cc:1196] T dee751ad1aca44f8b8089e50fe97e7ee P 7415a5be86774cc59fd6f269433d6a2b: Deleting WAL directory at /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704375975685-392-0/minicluster-data/ts-2/wal/wals/dee751ad1aca44f8b8089e50fe97e7ee
I20260929 17:53:06.457166 2039 tablet_replica.cc:330] T dee751ad1aca44f8b8089e50fe97e7ee P 0ff660fa6d574e1a83e0ccc53672f485: stopping tablet replica
I20260929 17:53:06.457381 2040 ts_tablet_manager.cc:1939] T dee751ad1aca44f8b8089e50fe97e7ee P 7415a5be86774cc59fd6f269433d6a2b: Deleting consensus metadata
I20260929 17:53:06.457525 2039 raft_consensus.cc:2229] T dee751ad1aca44f8b8089e50fe97e7ee P 0ff660fa6d574e1a83e0ccc53672f485 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260929 17:53:06.458384 1994 tablet.cc:2369] T 47dd32a1c7e44fe5882f0dc5aa078ce2 P 7415a5be86774cc59fd6f269433d6a2b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 17:53:06.458745 1497 catalog_manager.cc:4841] TS 7415a5be86774cc59fd6f269433d6a2b (127.0.98.3:38471): tablet dee751ad1aca44f8b8089e50fe97e7ee (table test-table [id=b396f96fcf894ffea40efa8e51b2a2f0]) successfully deleted
I20260929 17:53:06.459200 2017 tablet_bootstrap.cc:492] T 47dd32a1c7e44fe5882f0dc5aa078ce2 P 0ff660fa6d574e1a83e0ccc53672f485: No bootstrap required, opened a new log
I20260929 17:53:06.459286 2017 ts_tablet_manager.cc:1397] T 47dd32a1c7e44fe5882f0dc5aa078ce2 P 0ff660fa6d574e1a83e0ccc53672f485: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260929 17:53:06.457615 2038 ts_tablet_manager.cc:1918] T dee751ad1aca44f8b8089e50fe97e7ee P 1914a6c9793a4ceca993848ecd50c415: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 17:53:06.459623 2038 log.cc:1196] T dee751ad1aca44f8b8089e50fe97e7ee P 1914a6c9793a4ceca993848ecd50c415: Deleting WAL directory at /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704375975685-392-0/minicluster-data/ts-1/wal/wals/dee751ad1aca44f8b8089e50fe97e7ee
I20260929 17:53:06.459939 2038 ts_tablet_manager.cc:1939] T dee751ad1aca44f8b8089e50fe97e7ee P 1914a6c9793a4ceca993848ecd50c415: Deleting consensus metadata
I20260929 17:53:06.459918 2017 raft_consensus.cc:348] T 47dd32a1c7e44fe5882f0dc5aa078ce2 P 0ff660fa6d574e1a83e0ccc53672f485 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ff660fa6d574e1a83e0ccc53672f485" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40623 } } peers { permanent_uuid: "1914a6c9793a4ceca993848ecd50c415" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42193 } } peers { permanent_uuid: "7415a5be86774cc59fd6f269433d6a2b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38471 } }
I20260929 17:53:06.460011 2017 raft_consensus.cc:374] T 47dd32a1c7e44fe5882f0dc5aa078ce2 P 0ff660fa6d574e1a83e0ccc53672f485 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:53:06.460045 2017 raft_consensus.cc:729] T 47dd32a1c7e44fe5882f0dc5aa078ce2 P 0ff660fa6d574e1a83e0ccc53672f485 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ff660fa6d574e1a83e0ccc53672f485, State: Initialized, Role: FOLLOWER
I20260929 17:53:06.460171 2017 consensus_queue.cc:260] T 47dd32a1c7e44fe5882f0dc5aa078ce2 P 0ff660fa6d574e1a83e0ccc53672f485 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ff660fa6d574e1a83e0ccc53672f485" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40623 } } peers { permanent_uuid: "1914a6c9793a4ceca993848ecd50c415" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42193 } } peers { permanent_uuid: "7415a5be86774cc59fd6f269433d6a2b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38471 } }
I20260929 17:53:06.460410 2017 ts_tablet_manager.cc:1428] T 47dd32a1c7e44fe5882f0dc5aa078ce2 P 0ff660fa6d574e1a83e0ccc53672f485: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260929 17:53:06.461302 2016 tablet_bootstrap.cc:492] T 47dd32a1c7e44fe5882f0dc5aa078ce2 P 1914a6c9793a4ceca993848ecd50c415: No bootstrap required, opened a new log
I20260929 17:53:06.461386 2016 ts_tablet_manager.cc:1397] T 47dd32a1c7e44fe5882f0dc5aa078ce2 P 1914a6c9793a4ceca993848ecd50c415: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20260929 17:53:06.461814 2016 raft_consensus.cc:348] T 47dd32a1c7e44fe5882f0dc5aa078ce2 P 1914a6c9793a4ceca993848ecd50c415 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ff660fa6d574e1a83e0ccc53672f485" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40623 } } peers { permanent_uuid: "1914a6c9793a4ceca993848ecd50c415" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42193 } } peers { permanent_uuid: "7415a5be86774cc59fd6f269433d6a2b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38471 } }
I20260929 17:53:06.461961 2016 raft_consensus.cc:374] T 47dd32a1c7e44fe5882f0dc5aa078ce2 P 1914a6c9793a4ceca993848ecd50c415 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:53:06.461999 2016 raft_consensus.cc:729] T 47dd32a1c7e44fe5882f0dc5aa078ce2 P 1914a6c9793a4ceca993848ecd50c415 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1914a6c9793a4ceca993848ecd50c415, State: Initialized, Role: FOLLOWER
I20260929 17:53:06.462132 2016 consensus_queue.cc:260] T 47dd32a1c7e44fe5882f0dc5aa078ce2 P 1914a6c9793a4ceca993848ecd50c415 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ff660fa6d574e1a83e0ccc53672f485" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40623 } } peers { permanent_uuid: "1914a6c9793a4ceca993848ecd50c415" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42193 } } peers { permanent_uuid: "7415a5be86774cc59fd6f269433d6a2b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38471 } }
I20260929 17:53:06.462371 2016 ts_tablet_manager.cc:1428] T 47dd32a1c7e44fe5882f0dc5aa078ce2 P 1914a6c9793a4ceca993848ecd50c415: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 17:53:06.463635 1497 catalog_manager.cc:4841] TS 1914a6c9793a4ceca993848ecd50c415 (127.0.98.2:42193): tablet dee751ad1aca44f8b8089e50fe97e7ee (table test-table [id=b396f96fcf894ffea40efa8e51b2a2f0]) successfully deleted
I20260929 17:53:06.465206 2039 raft_consensus.cc:2258] T dee751ad1aca44f8b8089e50fe97e7ee P 0ff660fa6d574e1a83e0ccc53672f485 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 17:53:06.466717 2039 ts_tablet_manager.cc:1905] T dee751ad1aca44f8b8089e50fe97e7ee P 0ff660fa6d574e1a83e0ccc53672f485: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 17:53:06.469450 2039 ts_tablet_manager.cc:1918] T dee751ad1aca44f8b8089e50fe97e7ee P 0ff660fa6d574e1a83e0ccc53672f485: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 17:53:06.469550 2039 log.cc:1196] T dee751ad1aca44f8b8089e50fe97e7ee P 0ff660fa6d574e1a83e0ccc53672f485: Deleting WAL directory at /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704375975685-392-0/minicluster-data/ts-0/wal/wals/dee751ad1aca44f8b8089e50fe97e7ee
I20260929 17:53:06.469847 2039 ts_tablet_manager.cc:1939] T dee751ad1aca44f8b8089e50fe97e7ee P 0ff660fa6d574e1a83e0ccc53672f485: Deleting consensus metadata
I20260929 17:53:06.471271 1497 catalog_manager.cc:4841] TS 0ff660fa6d574e1a83e0ccc53672f485 (127.0.98.1:40623): tablet dee751ad1aca44f8b8089e50fe97e7ee (table test-table [id=b396f96fcf894ffea40efa8e51b2a2f0]) successfully deleted
W20260929 17:53:06.498890 1854 tablet.cc:2369] T 47dd32a1c7e44fe5882f0dc5aa078ce2 P 1914a6c9793a4ceca993848ecd50c415: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 17:53:06.611500 2024 raft_consensus.cc:482] T 47dd32a1c7e44fe5882f0dc5aa078ce2 P 7415a5be86774cc59fd6f269433d6a2b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 17:53:06.611640 2024 raft_consensus.cc:504] T 47dd32a1c7e44fe5882f0dc5aa078ce2 P 7415a5be86774cc59fd6f269433d6a2b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ff660fa6d574e1a83e0ccc53672f485" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40623 } } peers { permanent_uuid: "1914a6c9793a4ceca993848ecd50c415" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42193 } } peers { permanent_uuid: "7415a5be86774cc59fd6f269433d6a2b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38471 } }
I20260929 17:53:06.612689 2024 leader_election.cc:288] T 47dd32a1c7e44fe5882f0dc5aa078ce2 P 7415a5be86774cc59fd6f269433d6a2b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0ff660fa6d574e1a83e0ccc53672f485 (127.0.98.1:40623), 1914a6c9793a4ceca993848ecd50c415 (127.0.98.2:42193)
I20260929 17:53:06.616858 1886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:06.612916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40623 (local address 127.0.98.3:55633)
0929 17:53:06.613050 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:06.613065 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 17:53:06.613156 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:53:06.613441 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:53:06.613444 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:53:06.613463 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:53:06.613580 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:06.613586 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:06.614518 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:06.614520 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:06.615701 (+ 1181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:06.615711 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:06.616476 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:06.616479 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:06.616554 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:06.616575 (+ 21us) client_negotiation.cc:769] Sending connection context
0929 17:53:06.616648 (+ 73us) client_negotiation.cc:240] Negotiation successful
0929 17:53:06.616724 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260929 17:53:06.616858 2015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:06.613041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55633 (local address 127.0.98.1:40623)
0929 17:53:06.613163 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:06.613167 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:06.613186 (+ 19us) server_negotiation.cc:407] Connection header received
0929 17:53:06.613276 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:06.613279 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:06.613329 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:06.613424 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:53:06.613674 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:06.614416 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:06.615823 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:06.616373 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:06.616473 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:06.616538 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0929 17:53:06.616671 (+ 133us) server_negotiation.cc:299] Negotiation successful
0929 17:53:06.616747 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260929 17:53:06.617056 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:06.613326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41759 (local address 127.0.98.2:42193)
0929 17:53:06.613440 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:06.613444 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:06.613559 (+ 115us) server_negotiation.cc:407] Connection header received
0929 17:53:06.613650 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:06.613653 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:06.613695 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:06.613791 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:53:06.614018 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:06.614652 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:06.615977 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:06.616530 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:06.616616 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:06.616671 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0929 17:53:06.616858 (+ 187us) server_negotiation.cc:299] Negotiation successful
0929 17:53:06.616920 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 17:53:06.617208 2044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:06.613208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42193 (local address 127.0.98.3:41759)
0929 17:53:06.613471 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:06.613483 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 17:53:06.613573 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:53:06.613813 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:53:06.613815 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:53:06.613829 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:53:06.613926 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:06.613930 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:06.614739 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:06.614742 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:06.615878 (+ 1136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:06.615885 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:06.616637 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:06.616640 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:06.616733 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:06.616753 (+ 20us) client_negotiation.cc:769] Sending connection context
0929 17:53:06.616834 (+ 81us) client_negotiation.cc:240] Negotiation successful
0929 17:53:06.616889 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":207,"mutex_wait_us":1,"thread_start_us":77,"threads_started":1}
I20260929 17:53:06.617375 1667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47dd32a1c7e44fe5882f0dc5aa078ce2" candidate_uuid: "7415a5be86774cc59fd6f269433d6a2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ff660fa6d574e1a83e0ccc53672f485" is_pre_election: true
I20260929 17:53:06.617525 1667 raft_consensus.cc:2454] T 47dd32a1c7e44fe5882f0dc5aa078ce2 P 0ff660fa6d574e1a83e0ccc53672f485 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7415a5be86774cc59fd6f269433d6a2b in term 0.
I20260929 17:53:06.617664 1807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47dd32a1c7e44fe5882f0dc5aa078ce2" candidate_uuid: "7415a5be86774cc59fd6f269433d6a2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1914a6c9793a4ceca993848ecd50c415" is_pre_election: true
I20260929 17:53:06.617877 1807 raft_consensus.cc:2454] T 47dd32a1c7e44fe5882f0dc5aa078ce2 P 1914a6c9793a4ceca993848ecd50c415 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7415a5be86774cc59fd6f269433d6a2b in term 0.
I20260929 17:53:06.617879 1881 leader_election.cc:302] T 47dd32a1c7e44fe5882f0dc5aa078ce2 P 7415a5be86774cc59fd6f269433d6a2b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ff660fa6d574e1a83e0ccc53672f485, 7415a5be86774cc59fd6f269433d6a2b; no voters:
I20260929 17:53:06.618111 2024 raft_consensus.cc:2781] T 47dd32a1c7e44fe5882f0dc5aa078ce2 P 7415a5be86774cc59fd6f269433d6a2b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 17:53:06.618172 2024 raft_consensus.cc:482] T 47dd32a1c7e44fe5882f0dc5aa078ce2 P 7415a5be86774cc59fd6f269433d6a2b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 17:53:06.618207 2024 raft_consensus.cc:3037] T 47dd32a1c7e44fe5882f0dc5aa078ce2 P 7415a5be86774cc59fd6f269433d6a2b [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:53:06.619176 2024 raft_consensus.cc:504] T 47dd32a1c7e44fe5882f0dc5aa078ce2 P 7415a5be86774cc59fd6f269433d6a2b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ff660fa6d574e1a83e0ccc53672f485" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40623 } } peers { permanent_uuid: "1914a6c9793a4ceca993848ecd50c415" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42193 } } peers { permanent_uuid: "7415a5be86774cc59fd6f269433d6a2b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38471 } }
I20260929 17:53:06.619475 2024 leader_election.cc:288] T 47dd32a1c7e44fe5882f0dc5aa078ce2 P 7415a5be86774cc59fd6f269433d6a2b [CANDIDATE]: Term 1 election: Requested vote from peers 0ff660fa6d574e1a83e0ccc53672f485 (127.0.98.1:40623), 1914a6c9793a4ceca993848ecd50c415 (127.0.98.2:42193)
I20260929 17:53:06.619822 1807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47dd32a1c7e44fe5882f0dc5aa078ce2" candidate_uuid: "7415a5be86774cc59fd6f269433d6a2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1914a6c9793a4ceca993848ecd50c415"
I20260929 17:53:06.619846 1667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47dd32a1c7e44fe5882f0dc5aa078ce2" candidate_uuid: "7415a5be86774cc59fd6f269433d6a2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ff660fa6d574e1a83e0ccc53672f485"
I20260929 17:53:06.619923 1807 raft_consensus.cc:3037] T 47dd32a1c7e44fe5882f0dc5aa078ce2 P 1914a6c9793a4ceca993848ecd50c415 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:53:06.619953 1667 raft_consensus.cc:3037] T 47dd32a1c7e44fe5882f0dc5aa078ce2 P 0ff660fa6d574e1a83e0ccc53672f485 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:53:06.620823 1667 raft_consensus.cc:2454] T 47dd32a1c7e44fe5882f0dc5aa078ce2 P 0ff660fa6d574e1a83e0ccc53672f485 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7415a5be86774cc59fd6f269433d6a2b in term 1.
I20260929 17:53:06.620883 1807 raft_consensus.cc:2454] T 47dd32a1c7e44fe5882f0dc5aa078ce2 P 1914a6c9793a4ceca993848ecd50c415 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7415a5be86774cc59fd6f269433d6a2b in term 1.
I20260929 17:53:06.621079 1881 leader_election.cc:302] T 47dd32a1c7e44fe5882f0dc5aa078ce2 P 7415a5be86774cc59fd6f269433d6a2b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ff660fa6d574e1a83e0ccc53672f485, 7415a5be86774cc59fd6f269433d6a2b; no voters:
I20260929 17:53:06.621217 2024 raft_consensus.cc:2781] T 47dd32a1c7e44fe5882f0dc5aa078ce2 P 7415a5be86774cc59fd6f269433d6a2b [term 1 FOLLOWER]: Leader election won for term 1
I20260929 17:53:06.621438 2024 raft_consensus.cc:686] T 47dd32a1c7e44fe5882f0dc5aa078ce2 P 7415a5be86774cc59fd6f269433d6a2b [term 1 LEADER]: Becoming Leader. State: Replica: 7415a5be86774cc59fd6f269433d6a2b, State: Running, Role: LEADER
I20260929 17:53:06.621686 2024 consensus_queue.cc:237] T 47dd32a1c7e44fe5882f0dc5aa078ce2 P 7415a5be86774cc59fd6f269433d6a2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ff660fa6d574e1a83e0ccc53672f485" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40623 } } peers { permanent_uuid: "1914a6c9793a4ceca993848ecd50c415" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42193 } } peers { permanent_uuid: "7415a5be86774cc59fd6f269433d6a2b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38471 } }
I20260929 17:53:06.624003 1510 catalog_manager.cc:5495] T 47dd32a1c7e44fe5882f0dc5aa078ce2 P 7415a5be86774cc59fd6f269433d6a2b reported cstate change: term changed from 0 to 1, leader changed from <none> to 7415a5be86774cc59fd6f269433d6a2b (127.0.98.3). New cstate: current_term: 1 leader_uuid: "7415a5be86774cc59fd6f269433d6a2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ff660fa6d574e1a83e0ccc53672f485" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40623 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1914a6c9793a4ceca993848ecd50c415" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7415a5be86774cc59fd6f269433d6a2b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38471 } health_report { overall_health: HEALTHY } } }
Sep 29 17:53:06 dist-test-slave-53nh krb5kdc[1465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704385, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 17:53:06.682519 2052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:06.674726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42603 (local address 127.0.0.1:36878)
0929 17:53:06.674997 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:06.675012 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 17:53:06.675110 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:53:06.675357 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:53:06.675359 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:53:06.675787 (+ 428us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:53:06.675880 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:06.675885 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:06.676588 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:06.676590 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:06.677223 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:06.677231 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:06.677611 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:06.677613 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:06.677672 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:06.678196 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:53:06.678207 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:53:06.679613 (+ 1406us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:53:06.681142 (+ 1529us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:53:06.681151 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:53:06.681155 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:53:06.681405 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:53:06.681635 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:53:06.681637 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:53:06.681639 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:53:06.681709 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:53:06.682034 (+ 325us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:53:06.682040 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:53:06.682151 (+ 111us) client_negotiation.cc:769] Sending connection context
0929 17:53:06.682245 (+ 94us) client_negotiation.cc:240] Negotiation successful
0929 17:53:06.682380 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":89,"threads_started":1}
I20260929 17:53:06.682592 1559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:06.674809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36878 (local address 127.0.98.62:42603)
0929 17:53:06.674950 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:06.674953 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:06.675073 (+ 120us) server_negotiation.cc:407] Connection header received
0929 17:53:06.675202 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:06.675205 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:06.675268 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:06.675333 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:53:06.675979 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:06.676502 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:06.677331 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:06.677534 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:06.677604 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:06.679751 (+ 2147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:53:06.679754 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:53:06.679756 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:53:06.679777 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:53:06.681042 (+ 1265us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:53:06.681490 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:53:06.681493 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:53:06.681495 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:53:06.681569 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:53:06.681767 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:53:06.681769 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:53:06.681770 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:53:06.681934 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:53:06.682023 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0929 17:53:06.682353 (+ 330us) server_negotiation.cc:299] Negotiation successful
0929 17:53:06.682454 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 17:53:06.686450 1510 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36878:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260929 17:53:06.687134 1510 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 17:53:06.691425 1787 tablet_service.cc:1467] Processing CreateTablet for tablet fbbf678f47584428a4e14859964eed78 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a658697fa0954579b257c2a1ea10b3ab]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 17:53:06.691414 1647 tablet_service.cc:1467] Processing CreateTablet for tablet fbbf678f47584428a4e14859964eed78 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a658697fa0954579b257c2a1ea10b3ab]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 17:53:06.691821 1787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fbbf678f47584428a4e14859964eed78. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:53:06.691823 1647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fbbf678f47584428a4e14859964eed78. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:53:06.694589 2016 tablet_bootstrap.cc:492] T fbbf678f47584428a4e14859964eed78 P 1914a6c9793a4ceca993848ecd50c415: Bootstrap starting.
I20260929 17:53:06.694664 2017 tablet_bootstrap.cc:492] T fbbf678f47584428a4e14859964eed78 P 0ff660fa6d574e1a83e0ccc53672f485: Bootstrap starting.
I20260929 17:53:06.694885 1927 tablet_service.cc:1467] Processing CreateTablet for tablet fbbf678f47584428a4e14859964eed78 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a658697fa0954579b257c2a1ea10b3ab]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 17:53:06.695158 1927 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fbbf678f47584428a4e14859964eed78. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:53:06.695487 2016 tablet_bootstrap.cc:654] T fbbf678f47584428a4e14859964eed78 P 1914a6c9793a4ceca993848ecd50c415: Neither blocks nor log segments found. Creating new log.
I20260929 17:53:06.695540 2017 tablet_bootstrap.cc:654] T fbbf678f47584428a4e14859964eed78 P 0ff660fa6d574e1a83e0ccc53672f485: Neither blocks nor log segments found. Creating new log.
I20260929 17:53:06.696544 2016 tablet_bootstrap.cc:492] T fbbf678f47584428a4e14859964eed78 P 1914a6c9793a4ceca993848ecd50c415: No bootstrap required, opened a new log
I20260929 17:53:06.696622 2016 ts_tablet_manager.cc:1397] T fbbf678f47584428a4e14859964eed78 P 1914a6c9793a4ceca993848ecd50c415: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 17:53:06.696820 2017 tablet_bootstrap.cc:492] T fbbf678f47584428a4e14859964eed78 P 0ff660fa6d574e1a83e0ccc53672f485: No bootstrap required, opened a new log
I20260929 17:53:06.696877 2017 ts_tablet_manager.cc:1397] T fbbf678f47584428a4e14859964eed78 P 0ff660fa6d574e1a83e0ccc53672f485: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260929 17:53:06.697032 2016 raft_consensus.cc:348] T fbbf678f47584428a4e14859964eed78 P 1914a6c9793a4ceca993848ecd50c415 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ff660fa6d574e1a83e0ccc53672f485" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40623 } } peers { permanent_uuid: "1914a6c9793a4ceca993848ecd50c415" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42193 } } peers { permanent_uuid: "7415a5be86774cc59fd6f269433d6a2b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38471 } }
I20260929 17:53:06.697136 2016 raft_consensus.cc:374] T fbbf678f47584428a4e14859964eed78 P 1914a6c9793a4ceca993848ecd50c415 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:53:06.697161 2016 raft_consensus.cc:729] T fbbf678f47584428a4e14859964eed78 P 1914a6c9793a4ceca993848ecd50c415 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1914a6c9793a4ceca993848ecd50c415, State: Initialized, Role: FOLLOWER
I20260929 17:53:06.697263 2017 raft_consensus.cc:348] T fbbf678f47584428a4e14859964eed78 P 0ff660fa6d574e1a83e0ccc53672f485 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ff660fa6d574e1a83e0ccc53672f485" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40623 } } peers { permanent_uuid: "1914a6c9793a4ceca993848ecd50c415" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42193 } } peers { permanent_uuid: "7415a5be86774cc59fd6f269433d6a2b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38471 } }
I20260929 17:53:06.697288 2016 consensus_queue.cc:260] T fbbf678f47584428a4e14859964eed78 P 1914a6c9793a4ceca993848ecd50c415 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ff660fa6d574e1a83e0ccc53672f485" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40623 } } peers { permanent_uuid: "1914a6c9793a4ceca993848ecd50c415" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42193 } } peers { permanent_uuid: "7415a5be86774cc59fd6f269433d6a2b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38471 } }
I20260929 17:53:06.697356 2017 raft_consensus.cc:374] T fbbf678f47584428a4e14859964eed78 P 0ff660fa6d574e1a83e0ccc53672f485 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:53:06.697378 2017 raft_consensus.cc:729] T fbbf678f47584428a4e14859964eed78 P 0ff660fa6d574e1a83e0ccc53672f485 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ff660fa6d574e1a83e0ccc53672f485, State: Initialized, Role: FOLLOWER
I20260929 17:53:06.697477 2017 consensus_queue.cc:260] T fbbf678f47584428a4e14859964eed78 P 0ff660fa6d574e1a83e0ccc53672f485 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ff660fa6d574e1a83e0ccc53672f485" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40623 } } peers { permanent_uuid: "1914a6c9793a4ceca993848ecd50c415" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42193 } } peers { permanent_uuid: "7415a5be86774cc59fd6f269433d6a2b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38471 } }
I20260929 17:53:06.697521 2026 tablet_replica.cc:439] T fbbf678f47584428a4e14859964eed78 P 1914a6c9793a4ceca993848ecd50c415: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ff660fa6d574e1a83e0ccc53672f485" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40623 } } peers { permanent_uuid: "1914a6c9793a4ceca993848ecd50c415" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42193 } } peers { permanent_uuid: "7415a5be86774cc59fd6f269433d6a2b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38471 } } }
I20260929 17:53:06.697649 2026 tablet_replica.cc:442] T fbbf678f47584428a4e14859964eed78 P 1914a6c9793a4ceca993848ecd50c415: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 17:53:06.697736 2021 tablet_replica.cc:439] T fbbf678f47584428a4e14859964eed78 P 0ff660fa6d574e1a83e0ccc53672f485: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ff660fa6d574e1a83e0ccc53672f485" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40623 } } peers { permanent_uuid: "1914a6c9793a4ceca993848ecd50c415" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42193 } } peers { permanent_uuid: "7415a5be86774cc59fd6f269433d6a2b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38471 } } }
I20260929 17:53:06.697844 2022 tablet_bootstrap.cc:492] T fbbf678f47584428a4e14859964eed78 P 7415a5be86774cc59fd6f269433d6a2b: Bootstrap starting.
I20260929 17:53:06.697877 2021 tablet_replica.cc:442] T fbbf678f47584428a4e14859964eed78 P 0ff660fa6d574e1a83e0ccc53672f485: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 17:53:06.697530 2016 ts_tablet_manager.cc:1428] T fbbf678f47584428a4e14859964eed78 P 1914a6c9793a4ceca993848ecd50c415: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 17:53:06.697751 2017 ts_tablet_manager.cc:1428] T fbbf678f47584428a4e14859964eed78 P 0ff660fa6d574e1a83e0ccc53672f485: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 17:53:06.698673 2022 tablet_bootstrap.cc:654] T fbbf678f47584428a4e14859964eed78 P 7415a5be86774cc59fd6f269433d6a2b: Neither blocks nor log segments found. Creating new log.
I20260929 17:53:06.699767 2022 tablet_bootstrap.cc:492] T fbbf678f47584428a4e14859964eed78 P 7415a5be86774cc59fd6f269433d6a2b: No bootstrap required, opened a new log
I20260929 17:53:06.699857 2022 ts_tablet_manager.cc:1397] T fbbf678f47584428a4e14859964eed78 P 7415a5be86774cc59fd6f269433d6a2b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20260929 17:53:06.700659 2022 raft_consensus.cc:348] T fbbf678f47584428a4e14859964eed78 P 7415a5be86774cc59fd6f269433d6a2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ff660fa6d574e1a83e0ccc53672f485" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40623 } } peers { permanent_uuid: "1914a6c9793a4ceca993848ecd50c415" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42193 } } peers { permanent_uuid: "7415a5be86774cc59fd6f269433d6a2b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38471 } }
I20260929 17:53:06.700795 2022 raft_consensus.cc:374] T fbbf678f47584428a4e14859964eed78 P 7415a5be86774cc59fd6f269433d6a2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:53:06.700840 2022 raft_consensus.cc:729] T fbbf678f47584428a4e14859964eed78 P 7415a5be86774cc59fd6f269433d6a2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7415a5be86774cc59fd6f269433d6a2b, State: Initialized, Role: FOLLOWER
I20260929 17:53:06.700958 2022 consensus_queue.cc:260] T fbbf678f47584428a4e14859964eed78 P 7415a5be86774cc59fd6f269433d6a2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ff660fa6d574e1a83e0ccc53672f485" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40623 } } peers { permanent_uuid: "1914a6c9793a4ceca993848ecd50c415" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42193 } } peers { permanent_uuid: "7415a5be86774cc59fd6f269433d6a2b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38471 } }
I20260929 17:53:06.701225 2024 tablet_replica.cc:439] T fbbf678f47584428a4e14859964eed78 P 7415a5be86774cc59fd6f269433d6a2b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ff660fa6d574e1a83e0ccc53672f485" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40623 } } peers { permanent_uuid: "1914a6c9793a4ceca993848ecd50c415" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42193 } } peers { permanent_uuid: "7415a5be86774cc59fd6f269433d6a2b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38471 } } }
I20260929 17:53:06.701350 2024 tablet_replica.cc:442] T fbbf678f47584428a4e14859964eed78 P 7415a5be86774cc59fd6f269433d6a2b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 17:53:06.701287 2022 ts_tablet_manager.cc:1428] T fbbf678f47584428a4e14859964eed78 P 7415a5be86774cc59fd6f269433d6a2b: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260929 17:53:06.938688 2021 raft_consensus.cc:482] T fbbf678f47584428a4e14859964eed78 P 0ff660fa6d574e1a83e0ccc53672f485 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 17:53:06.938828 2021 raft_consensus.cc:504] T fbbf678f47584428a4e14859964eed78 P 0ff660fa6d574e1a83e0ccc53672f485 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ff660fa6d574e1a83e0ccc53672f485" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40623 } } peers { permanent_uuid: "1914a6c9793a4ceca993848ecd50c415" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42193 } } peers { permanent_uuid: "7415a5be86774cc59fd6f269433d6a2b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38471 } }
I20260929 17:53:06.939945 2021 leader_election.cc:288] T fbbf678f47584428a4e14859964eed78 P 0ff660fa6d574e1a83e0ccc53672f485 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1914a6c9793a4ceca993848ecd50c415 (127.0.98.2:42193), 7415a5be86774cc59fd6f269433d6a2b (127.0.98.3:38471)
I20260929 17:53:06.943826 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:06.940218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34401 (local address 127.0.98.2:42193)
0929 17:53:06.940418 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:06.940423 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:06.940544 (+ 121us) server_negotiation.cc:407] Connection header received
0929 17:53:06.940679 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:06.940682 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:06.940734 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:06.940813 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:53:06.941118 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:06.941713 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:06.942903 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:06.943302 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:06.943399 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:06.943458 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0929 17:53:06.943625 (+ 167us) server_negotiation.cc:299] Negotiation successful
0929 17:53:06.943701 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106}
I20260929 17:53:06.943866 2056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:06.940162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42193 (local address 127.0.98.1:34401)
0929 17:53:06.940437 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:06.940451 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 17:53:06.940579 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:53:06.940851 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:53:06.940855 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:53:06.940878 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:53:06.941020 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:06.941027 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:06.941811 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:06.941814 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:06.942787 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:06.942798 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:06.943417 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:06.943421 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:06.943510 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:06.943535 (+ 25us) client_negotiation.cc:769] Sending connection context
0929 17:53:06.943602 (+ 67us) client_negotiation.cc:240] Negotiation successful
0929 17:53:06.943679 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":208,"mutex_wait_us":1,"spinlock_wait_cycles":5376,"thread_start_us":109,"threads_started":1}
I20260929 17:53:06.944097 2058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:06.940670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51949 (local address 127.0.98.3:38471)
0929 17:53:06.940887 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:06.940890 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:06.940904 (+ 14us) server_negotiation.cc:407] Connection header received
0929 17:53:06.941033 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:06.941036 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:06.941085 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:06.941158 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:53:06.941386 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:06.941944 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:06.942962 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:06.943436 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:06.943527 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:06.943589 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0929 17:53:06.943832 (+ 243us) server_negotiation.cc:299] Negotiation successful
0929 17:53:06.943929 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":70,"threads_started":1}
I20260929 17:53:06.944267 1807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbbf678f47584428a4e14859964eed78" candidate_uuid: "0ff660fa6d574e1a83e0ccc53672f485" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1914a6c9793a4ceca993848ecd50c415" is_pre_election: true
I20260929 17:53:06.944352 2057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:06.940532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38471 (local address 127.0.98.1:51949)
0929 17:53:06.940825 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:06.940838 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 17:53:06.940907 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:53:06.941179 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:53:06.941182 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:53:06.941199 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:53:06.941282 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:06.941289 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:06.942063 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:06.942066 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:06.942839 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:06.942849 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:06.943560 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:06.943562 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:06.943654 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:06.943677 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 17:53:06.943745 (+ 68us) client_negotiation.cc:240] Negotiation successful
0929 17:53:06.943813 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":56,"threads_started":1}
I20260929 17:53:06.944420 1807 raft_consensus.cc:2454] T fbbf678f47584428a4e14859964eed78 P 1914a6c9793a4ceca993848ecd50c415 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ff660fa6d574e1a83e0ccc53672f485 in term 0.
I20260929 17:53:06.944726 1601 leader_election.cc:302] T fbbf678f47584428a4e14859964eed78 P 0ff660fa6d574e1a83e0ccc53672f485 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ff660fa6d574e1a83e0ccc53672f485, 1914a6c9793a4ceca993848ecd50c415; no voters:
I20260929 17:53:06.944743 1947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbbf678f47584428a4e14859964eed78" candidate_uuid: "0ff660fa6d574e1a83e0ccc53672f485" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7415a5be86774cc59fd6f269433d6a2b" is_pre_election: true
I20260929 17:53:06.944861 1947 raft_consensus.cc:2454] T fbbf678f47584428a4e14859964eed78 P 7415a5be86774cc59fd6f269433d6a2b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ff660fa6d574e1a83e0ccc53672f485 in term 0.
I20260929 17:53:06.944942 2021 raft_consensus.cc:2781] T fbbf678f47584428a4e14859964eed78 P 0ff660fa6d574e1a83e0ccc53672f485 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 17:53:06.944990 2021 raft_consensus.cc:482] T fbbf678f47584428a4e14859964eed78 P 0ff660fa6d574e1a83e0ccc53672f485 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 17:53:06.945047 2021 raft_consensus.cc:3037] T fbbf678f47584428a4e14859964eed78 P 0ff660fa6d574e1a83e0ccc53672f485 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:53:06.945964 2021 raft_consensus.cc:504] T fbbf678f47584428a4e14859964eed78 P 0ff660fa6d574e1a83e0ccc53672f485 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ff660fa6d574e1a83e0ccc53672f485" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40623 } } peers { permanent_uuid: "1914a6c9793a4ceca993848ecd50c415" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42193 } } peers { permanent_uuid: "7415a5be86774cc59fd6f269433d6a2b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38471 } }
I20260929 17:53:06.946244 2021 leader_election.cc:288] T fbbf678f47584428a4e14859964eed78 P 0ff660fa6d574e1a83e0ccc53672f485 [CANDIDATE]: Term 1 election: Requested vote from peers 1914a6c9793a4ceca993848ecd50c415 (127.0.98.2:42193), 7415a5be86774cc59fd6f269433d6a2b (127.0.98.3:38471)
I20260929 17:53:06.946552 1807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbbf678f47584428a4e14859964eed78" candidate_uuid: "0ff660fa6d574e1a83e0ccc53672f485" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1914a6c9793a4ceca993848ecd50c415"
I20260929 17:53:06.946573 1947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbbf678f47584428a4e14859964eed78" candidate_uuid: "0ff660fa6d574e1a83e0ccc53672f485" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7415a5be86774cc59fd6f269433d6a2b"
I20260929 17:53:06.946668 1807 raft_consensus.cc:3037] T fbbf678f47584428a4e14859964eed78 P 1914a6c9793a4ceca993848ecd50c415 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:53:06.946671 1947 raft_consensus.cc:3037] T fbbf678f47584428a4e14859964eed78 P 7415a5be86774cc59fd6f269433d6a2b [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:53:06.947479 1947 raft_consensus.cc:2454] T fbbf678f47584428a4e14859964eed78 P 7415a5be86774cc59fd6f269433d6a2b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ff660fa6d574e1a83e0ccc53672f485 in term 1.
I20260929 17:53:06.947525 1807 raft_consensus.cc:2454] T fbbf678f47584428a4e14859964eed78 P 1914a6c9793a4ceca993848ecd50c415 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ff660fa6d574e1a83e0ccc53672f485 in term 1.
I20260929 17:53:06.947706 1601 leader_election.cc:302] T fbbf678f47584428a4e14859964eed78 P 0ff660fa6d574e1a83e0ccc53672f485 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ff660fa6d574e1a83e0ccc53672f485, 7415a5be86774cc59fd6f269433d6a2b; no voters:
I20260929 17:53:06.947852 2021 raft_consensus.cc:2781] T fbbf678f47584428a4e14859964eed78 P 0ff660fa6d574e1a83e0ccc53672f485 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 17:53:06.948019 2021 raft_consensus.cc:686] T fbbf678f47584428a4e14859964eed78 P 0ff660fa6d574e1a83e0ccc53672f485 [term 1 LEADER]: Becoming Leader. State: Replica: 0ff660fa6d574e1a83e0ccc53672f485, State: Running, Role: LEADER
I20260929 17:53:06.948257 2021 consensus_queue.cc:237] T fbbf678f47584428a4e14859964eed78 P 0ff660fa6d574e1a83e0ccc53672f485 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ff660fa6d574e1a83e0ccc53672f485" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40623 } } peers { permanent_uuid: "1914a6c9793a4ceca993848ecd50c415" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42193 } } peers { permanent_uuid: "7415a5be86774cc59fd6f269433d6a2b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38471 } }
I20260929 17:53:06.949573 2059 tablet_replica.cc:439] T fbbf678f47584428a4e14859964eed78 P 0ff660fa6d574e1a83e0ccc53672f485: TxnStatusTablet state changed. Reason: New leader 0ff660fa6d574e1a83e0ccc53672f485. Latest consensus state: current_term: 1 leader_uuid: "0ff660fa6d574e1a83e0ccc53672f485" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ff660fa6d574e1a83e0ccc53672f485" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40623 } } peers { permanent_uuid: "1914a6c9793a4ceca993848ecd50c415" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42193 } } peers { permanent_uuid: "7415a5be86774cc59fd6f269433d6a2b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38471 } } }
I20260929 17:53:06.949676 2059 tablet_replica.cc:442] T fbbf678f47584428a4e14859964eed78 P 0ff660fa6d574e1a83e0ccc53672f485: This TxnStatusTablet replica's current role is: LEADER
I20260929 17:53:06.949927 2061 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260929 17:53:06.950076 2061 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260929 17:53:06.950703 1509 catalog_manager.cc:5495] T fbbf678f47584428a4e14859964eed78 P 0ff660fa6d574e1a83e0ccc53672f485 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0ff660fa6d574e1a83e0ccc53672f485 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "0ff660fa6d574e1a83e0ccc53672f485" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ff660fa6d574e1a83e0ccc53672f485" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40623 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1914a6c9793a4ceca993848ecd50c415" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7415a5be86774cc59fd6f269433d6a2b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38471 } health_report { overall_health: UNKNOWN } } }
I20260929 17:53:06.994415 2052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:06.991016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40623 (local address 127.0.0.1:35480)
0929 17:53:06.991155 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:06.991171 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 17:53:06.991282 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:53:06.991581 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:53:06.991584 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:53:06.991594 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 17:53:06.991689 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:06.991694 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:06.992367 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:06.992371 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:06.992961 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:06.992969 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:06.993387 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:06.993390 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:06.993469 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:06.993510 (+ 41us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 17:53:06.994132 (+ 622us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 17:53:06.994146 (+ 14us) client_negotiation.cc:769] Sending connection context
0929 17:53:06.994208 (+ 62us) client_negotiation.cc:240] Negotiation successful
0929 17:53:06.994304 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260929 17:53:06.994407 2015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:06.991156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35480 (local address 127.0.98.1:40623)
0929 17:53:06.991290 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:06.991294 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:06.991308 (+ 14us) server_negotiation.cc:407] Connection header received
0929 17:53:06.991430 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:06.991433 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:06.991483 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:06.991561 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 17:53:06.991790 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:06.992273 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:06.993083 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:06.993295 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:06.993380 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:06.993635 (+ 255us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 17:53:06.994047 (+ 412us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 17:53:06.994143 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0929 17:53:06.994216 (+ 73us) server_negotiation.cc:299] Negotiation successful
0929 17:53:06.994268 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 17:53:06.999359 2024 consensus_queue.cc:1035] T 47dd32a1c7e44fe5882f0dc5aa078ce2 P 7415a5be86774cc59fd6f269433d6a2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ff660fa6d574e1a83e0ccc53672f485" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 17:53:07.002254 1807 raft_consensus.cc:1261] T fbbf678f47584428a4e14859964eed78 P 1914a6c9793a4ceca993848ecd50c415 [term 1 FOLLOWER]: Refusing update from remote peer 0ff660fa6d574e1a83e0ccc53672f485: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 17:53:07.003171 1947 raft_consensus.cc:1261] T fbbf678f47584428a4e14859964eed78 P 7415a5be86774cc59fd6f269433d6a2b [term 1 FOLLOWER]: Refusing update from remote peer 0ff660fa6d574e1a83e0ccc53672f485: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 17:53:07.003540 2021 consensus_queue.cc:1035] T fbbf678f47584428a4e14859964eed78 P 0ff660fa6d574e1a83e0ccc53672f485 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1914a6c9793a4ceca993848ecd50c415" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 17:53:07.003918 2021 consensus_queue.cc:1035] T fbbf678f47584428a4e14859964eed78 P 0ff660fa6d574e1a83e0ccc53672f485 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7415a5be86774cc59fd6f269433d6a2b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 17:53:07.004289 2024 consensus_queue.cc:1035] T 47dd32a1c7e44fe5882f0dc5aa078ce2 P 7415a5be86774cc59fd6f269433d6a2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1914a6c9793a4ceca993848ecd50c415" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 17:53:07.006321 2026 tablet_replica.cc:439] T fbbf678f47584428a4e14859964eed78 P 1914a6c9793a4ceca993848ecd50c415: TxnStatusTablet state changed. Reason: New leader 0ff660fa6d574e1a83e0ccc53672f485. Latest consensus state: current_term: 1 leader_uuid: "0ff660fa6d574e1a83e0ccc53672f485" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ff660fa6d574e1a83e0ccc53672f485" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40623 } } peers { permanent_uuid: "1914a6c9793a4ceca993848ecd50c415" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42193 } } peers { permanent_uuid: "7415a5be86774cc59fd6f269433d6a2b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38471 } } }
I20260929 17:53:07.006433 2026 tablet_replica.cc:442] T fbbf678f47584428a4e14859964eed78 P 1914a6c9793a4ceca993848ecd50c415: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 17:53:07.010061 2024 tablet_replica.cc:439] T fbbf678f47584428a4e14859964eed78 P 7415a5be86774cc59fd6f269433d6a2b: TxnStatusTablet state changed. Reason: New leader 0ff660fa6d574e1a83e0ccc53672f485. Latest consensus state: current_term: 1 leader_uuid: "0ff660fa6d574e1a83e0ccc53672f485" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ff660fa6d574e1a83e0ccc53672f485" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40623 } } peers { permanent_uuid: "1914a6c9793a4ceca993848ecd50c415" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42193 } } peers { permanent_uuid: "7415a5be86774cc59fd6f269433d6a2b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38471 } } }
I20260929 17:53:07.010190 2024 tablet_replica.cc:442] T fbbf678f47584428a4e14859964eed78 P 7415a5be86774cc59fd6f269433d6a2b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 17:53:07.011656 2021 tablet_replica.cc:439] T fbbf678f47584428a4e14859964eed78 P 0ff660fa6d574e1a83e0ccc53672f485: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0ff660fa6d574e1a83e0ccc53672f485" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ff660fa6d574e1a83e0ccc53672f485" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40623 } } peers { permanent_uuid: "1914a6c9793a4ceca993848ecd50c415" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42193 } } peers { permanent_uuid: "7415a5be86774cc59fd6f269433d6a2b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38471 } } }
I20260929 17:53:07.011775 2021 tablet_replica.cc:442] T fbbf678f47584428a4e14859964eed78 P 0ff660fa6d574e1a83e0ccc53672f485: This TxnStatusTablet replica's current role is: LEADER
I20260929 17:53:07.012068 2024 tablet_replica.cc:439] T fbbf678f47584428a4e14859964eed78 P 7415a5be86774cc59fd6f269433d6a2b: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0ff660fa6d574e1a83e0ccc53672f485" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ff660fa6d574e1a83e0ccc53672f485" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40623 } } peers { permanent_uuid: "1914a6c9793a4ceca993848ecd50c415" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42193 } } peers { permanent_uuid: "7415a5be86774cc59fd6f269433d6a2b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38471 } } }
I20260929 17:53:07.012162 2024 tablet_replica.cc:442] T fbbf678f47584428a4e14859964eed78 P 7415a5be86774cc59fd6f269433d6a2b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 17:53:07.014797 2075 tablet_replica.cc:439] T fbbf678f47584428a4e14859964eed78 P 0ff660fa6d574e1a83e0ccc53672f485: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0ff660fa6d574e1a83e0ccc53672f485" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ff660fa6d574e1a83e0ccc53672f485" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40623 } } peers { permanent_uuid: "1914a6c9793a4ceca993848ecd50c415" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42193 } } peers { permanent_uuid: "7415a5be86774cc59fd6f269433d6a2b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38471 } } }
I20260929 17:53:07.014915 2075 tablet_replica.cc:442] T fbbf678f47584428a4e14859964eed78 P 0ff660fa6d574e1a83e0ccc53672f485: This TxnStatusTablet replica's current role is: LEADER
I20260929 17:53:07.017061 2069 tablet_replica.cc:439] T fbbf678f47584428a4e14859964eed78 P 1914a6c9793a4ceca993848ecd50c415: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0ff660fa6d574e1a83e0ccc53672f485" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ff660fa6d574e1a83e0ccc53672f485" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40623 } } peers { permanent_uuid: "1914a6c9793a4ceca993848ecd50c415" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42193 } } peers { permanent_uuid: "7415a5be86774cc59fd6f269433d6a2b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38471 } } }
I20260929 17:53:07.017170 2069 tablet_replica.cc:442] T fbbf678f47584428a4e14859964eed78 P 1914a6c9793a4ceca993848ecd50c415: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 17:53:07.024397 2058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:07.019251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52140 (local address 127.0.98.3:38471)
0929 17:53:07.019393 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:07.019398 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:07.019571 (+ 173us) server_negotiation.cc:407] Connection header received
0929 17:53:07.019788 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:07.019792 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:07.019859 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:07.019940 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 17:53:07.020347 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:07.021164 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:07.022442 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:07.022797 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:07.022873 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:07.023290 (+ 417us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 17:53:07.023913 (+ 623us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 17:53:07.023999 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0929 17:53:07.024172 (+ 173us) server_negotiation.cc:299] Negotiation successful
0929 17:53:07.024253 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 17:53:07.040738 2015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:07.033205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53401 (local address 127.0.98.1:40623)
0929 17:53:07.033411 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:07.033415 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:07.037398 (+ 3983us) server_negotiation.cc:407] Connection header received
0929 17:53:07.037531 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:07.037533 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:07.037590 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:07.037660 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 17:53:07.037917 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:07.038441 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:07.039691 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:07.039912 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:07.040000 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:07.040327 (+ 327us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 17:53:07.040404 (+ 77us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 17:53:07.040459 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0929 17:53:07.040565 (+ 106us) server_negotiation.cc:299] Negotiation successful
0929 17:53:07.040616 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122}
I20260929 17:53:07.040807 2082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:07.033100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40623 (local address 127.0.98.3:53401)
0929 17:53:07.037284 (+ 4184us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:07.037302 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 17:53:07.037439 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:53:07.037681 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:53:07.037684 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:53:07.037692 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 17:53:07.037828 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:07.037834 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:07.038528 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:07.038531 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:07.039562 (+ 1031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:07.039572 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:07.040025 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:07.040028 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:07.040119 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:07.040177 (+ 58us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 17:53:07.040480 (+ 303us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 17:53:07.040486 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 17:53:07.040551 (+ 65us) client_negotiation.cc:240] Negotiation successful
0929 17:53:07.040634 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4049,"thread_start_us":95,"threads_started":1}
I20260929 17:53:07.067229 2058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:07.063414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55737 (local address 127.0.98.3:38471)
0929 17:53:07.063592 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:07.063605 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:07.063749 (+ 144us) server_negotiation.cc:407] Connection header received
0929 17:53:07.063858 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:07.063862 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:07.063914 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:07.063988 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 17:53:07.064222 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:07.064977 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:07.066267 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:07.066484 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:07.066561 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:07.066872 (+ 311us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 17:53:07.066936 (+ 64us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 17:53:07.066982 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0929 17:53:07.067080 (+ 98us) server_negotiation.cc:299] Negotiation successful
0929 17:53:07.067121 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 17:53:07.067341 2086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:07.063329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38471 (local address 127.0.98.1:55737)
0929 17:53:07.063672 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:07.063685 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 17:53:07.063779 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:53:07.064003 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:53:07.064005 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:53:07.064010 (+ 5us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 17:53:07.064120 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:07.064124 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:07.065147 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:07.065151 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:07.066153 (+ 1002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:07.066164 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:07.066604 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:07.066608 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:07.066703 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:07.066763 (+ 60us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 17:53:07.067007 (+ 244us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 17:53:07.067013 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 17:53:07.067075 (+ 62us) client_negotiation.cc:240] Negotiation successful
0929 17:53:07.067155 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":99,"threads_started":1}
I20260929 17:53:07.091394 1510 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36876:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 17:53:07.091576 1510 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36876:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 17:53:07.093891 1510 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bf0bd8ed30e946f49cc31cdf2f3f5424: Sending DeleteTablet for 3 replicas of tablet 47dd32a1c7e44fe5882f0dc5aa078ce2
I20260929 17:53:07.094532 1787 tablet_service.cc:1514] Processing DeleteTablet for tablet 47dd32a1c7e44fe5882f0dc5aa078ce2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 17:53:07 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54864
I20260929 17:53:07.094740 1927 tablet_service.cc:1514] Processing DeleteTablet for tablet 47dd32a1c7e44fe5882f0dc5aa078ce2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 17:53:07 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52134
I20260929 17:53:07.095095 2090 tablet_replica.cc:330] T 47dd32a1c7e44fe5882f0dc5aa078ce2 P 7415a5be86774cc59fd6f269433d6a2b: stopping tablet replica
I20260929 17:53:07.095158 1647 tablet_service.cc:1514] Processing DeleteTablet for tablet 47dd32a1c7e44fe5882f0dc5aa078ce2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 17:53:07 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35466
I20260929 17:53:07.095224 2090 raft_consensus.cc:2229] T 47dd32a1c7e44fe5882f0dc5aa078ce2 P 7415a5be86774cc59fd6f269433d6a2b [term 1 LEADER]: Raft consensus shutting down.
I20260929 17:53:07.095530 2090 raft_consensus.cc:2258] T 47dd32a1c7e44fe5882f0dc5aa078ce2 P 7415a5be86774cc59fd6f269433d6a2b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 17:53:07.096225 2090 ts_tablet_manager.cc:1905] T 47dd32a1c7e44fe5882f0dc5aa078ce2 P 7415a5be86774cc59fd6f269433d6a2b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 17:53:07.096642 2091 tablet_replica.cc:330] T 47dd32a1c7e44fe5882f0dc5aa078ce2 P 0ff660fa6d574e1a83e0ccc53672f485: stopping tablet replica
I20260929 17:53:07.097297 2091 raft_consensus.cc:2229] T 47dd32a1c7e44fe5882f0dc5aa078ce2 P 0ff660fa6d574e1a83e0ccc53672f485 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 17:53:07.097805 2091 raft_consensus.cc:2258] T 47dd32a1c7e44fe5882f0dc5aa078ce2 P 0ff660fa6d574e1a83e0ccc53672f485 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 17:53:07.098567 2091 ts_tablet_manager.cc:1905] T 47dd32a1c7e44fe5882f0dc5aa078ce2 P 0ff660fa6d574e1a83e0ccc53672f485: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 17:53:07.099236 2090 ts_tablet_manager.cc:1918] T 47dd32a1c7e44fe5882f0dc5aa078ce2 P 7415a5be86774cc59fd6f269433d6a2b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 17:53:07.099313 2090 log.cc:1196] T 47dd32a1c7e44fe5882f0dc5aa078ce2 P 7415a5be86774cc59fd6f269433d6a2b: Deleting WAL directory at /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704375975685-392-0/minicluster-data/ts-2/wal/wals/47dd32a1c7e44fe5882f0dc5aa078ce2
I20260929 17:53:07.099553 2090 ts_tablet_manager.cc:1939] T 47dd32a1c7e44fe5882f0dc5aa078ce2 P 7415a5be86774cc59fd6f269433d6a2b: Deleting consensus metadata
I20260929 17:53:07.100418 1497 catalog_manager.cc:4841] TS 7415a5be86774cc59fd6f269433d6a2b (127.0.98.3:38471): tablet 47dd32a1c7e44fe5882f0dc5aa078ce2 (table test-table [id=8da611cccdd948c2ab776776371ec16f]) successfully deleted
I20260929 17:53:07.102346 2091 ts_tablet_manager.cc:1918] T 47dd32a1c7e44fe5882f0dc5aa078ce2 P 0ff660fa6d574e1a83e0ccc53672f485: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 17:53:07.102429 2091 log.cc:1196] T 47dd32a1c7e44fe5882f0dc5aa078ce2 P 0ff660fa6d574e1a83e0ccc53672f485: Deleting WAL directory at /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704375975685-392-0/minicluster-data/ts-0/wal/wals/47dd32a1c7e44fe5882f0dc5aa078ce2
I20260929 17:53:07.102656 2091 ts_tablet_manager.cc:1939] T 47dd32a1c7e44fe5882f0dc5aa078ce2 P 0ff660fa6d574e1a83e0ccc53672f485: Deleting consensus metadata
I20260929 17:53:07.103377 2089 tablet_replica.cc:330] T 47dd32a1c7e44fe5882f0dc5aa078ce2 P 1914a6c9793a4ceca993848ecd50c415: stopping tablet replica
I20260929 17:53:07.103403 1497 catalog_manager.cc:4841] TS 0ff660fa6d574e1a83e0ccc53672f485 (127.0.98.1:40623): tablet 47dd32a1c7e44fe5882f0dc5aa078ce2 (table test-table [id=8da611cccdd948c2ab776776371ec16f]) successfully deleted
I20260929 17:53:07.103510 2089 raft_consensus.cc:2229] T 47dd32a1c7e44fe5882f0dc5aa078ce2 P 1914a6c9793a4ceca993848ecd50c415 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 17:53:07.103627 2089 raft_consensus.cc:2258] T 47dd32a1c7e44fe5882f0dc5aa078ce2 P 1914a6c9793a4ceca993848ecd50c415 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 17:53:07.104308 2089 ts_tablet_manager.cc:1905] T 47dd32a1c7e44fe5882f0dc5aa078ce2 P 1914a6c9793a4ceca993848ecd50c415: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 17:53:07.107213 2089 ts_tablet_manager.cc:1918] T 47dd32a1c7e44fe5882f0dc5aa078ce2 P 1914a6c9793a4ceca993848ecd50c415: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 17:53:07.107311 2089 log.cc:1196] T 47dd32a1c7e44fe5882f0dc5aa078ce2 P 1914a6c9793a4ceca993848ecd50c415: Deleting WAL directory at /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790704375975685-392-0/minicluster-data/ts-1/wal/wals/47dd32a1c7e44fe5882f0dc5aa078ce2
I20260929 17:53:07.107542 2089 ts_tablet_manager.cc:1939] T 47dd32a1c7e44fe5882f0dc5aa078ce2 P 1914a6c9793a4ceca993848ecd50c415: Deleting consensus metadata
I20260929 17:53:07.108309 1497 catalog_manager.cc:4841] TS 1914a6c9793a4ceca993848ecd50c415 (127.0.98.2:42193): tablet 47dd32a1c7e44fe5882f0dc5aa078ce2 (table test-table [id=8da611cccdd948c2ab776776371ec16f]) successfully deleted
Sep 29 17:53:07 dist-test-slave-53nh krb5kdc[1465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704386, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260929 17:53:07.117767 2015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:07.097089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35494 (local address 127.0.98.1:40623)
0929 17:53:07.097199 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:07.097203 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:07.105347 (+ 8144us) server_negotiation.cc:407] Connection header received
0929 17:53:07.105544 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:07.105547 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:07.105597 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:07.105645 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:53:07.106601 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:07.107391 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:07.109287 (+ 1896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:07.109632 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:07.109769 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:07.113641 (+ 3872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:53:07.113653 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:53:07.113673 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:53:07.113708 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:53:07.115659 (+ 1951us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:53:07.116154 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:53:07.116161 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:53:07.116163 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:53:07.116254 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:53:07.116537 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:53:07.116540 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:53:07.116542 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:53:07.116851 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:53:07.117090 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0929 17:53:07.117309 (+ 219us) server_negotiation.cc:299] Negotiation successful
0929 17:53:07.117587 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
W20260929 17:53:07.118175 1707 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35494
I20260929 17:53:07.127534 1559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:07.119975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36880 (local address 127.0.98.62:42603)
0929 17:53:07.120108 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:07.120112 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:07.120543 (+ 431us) server_negotiation.cc:407] Connection header received
0929 17:53:07.120650 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:07.120653 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:07.120705 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:07.120779 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:53:07.121518 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:07.122032 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:07.123097 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:07.123302 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:07.123371 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:07.124796 (+ 1425us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:53:07.124799 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:53:07.124801 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:53:07.124824 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:53:07.126150 (+ 1326us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:53:07.126559 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:53:07.126561 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:53:07.126563 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:53:07.126629 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:53:07.126860 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:53:07.126862 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:53:07.126863 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:53:07.127016 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:53:07.127104 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0929 17:53:07.127311 (+ 207us) server_negotiation.cc:299] Negotiation successful
0929 17:53:07.127400 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
W20260929 17:53:07.128029 1510 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36880
I20260929 17:53:07.129436 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskup3Soe/build/debug/bin/kudu with pid 1582
I20260929 17:53:07.139369 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskup3Soe/build/debug/bin/kudu with pid 1722
I20260929 17:53:07.145810 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskup3Soe/build/debug/bin/kudu with pid 1862
I20260929 17:53:07.150647 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskup3Soe/build/debug/bin/kudu with pid 1478
2026-09-29T17:53:07Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (6029 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704375975685-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 17:53:07 dist-test-slave-53nh krb5kdc[2098](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 17:53:07 dist-test-slave-53nh krb5kdc[2098](info): set up 2 sockets
Sep 29 17:53:07 dist-test-slave-53nh krb5kdc[2098](info): commencing operation
krb5kdc: starting...
W20260929 17:53:08.104592 392 mini_kdc.cc:121] Time spent starting KDC: real 0.933s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704375975685-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704375975685-392-0/krb5kdc/test-user.keytab.
Sep 29 17:53:08 dist-test-slave-53nh krb5kdc[2098](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704388, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T17:53:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T17:53:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704375975685-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704375975685-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 17:53:08.214071 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskup3Soe/build/debug/bin/kudu
/tmp/dist-test-taskup3Soe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704375975685-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704375975685-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704375975685-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704375975685-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42197
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704375975685-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42197
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704375975685-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704375975685-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704375975685-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:53:08.320705 2111 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:53:08.320976 2111 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:53:08.321059 2111 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:53:08.325035 2111 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 17:53:08.325095 2111 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260929 17:53:08.325119 2111 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:53:08.325140 2111 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:53:08.325173 2111 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 17:53:08.325202 2111 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 17:53:08.330451 2111 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704375975685-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704375975685-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42197
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704375975685-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42197
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704375975685-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2111
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704375975685-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:53:08.331486 2111 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:53:08.332468 2111 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 17:53:08.340663 2111 server_base.cc:1034] running on GCE node
W20260929 17:53:08.340728 2116 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:53:08.340667 2119 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:53:08.340860 2117 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:53:08.341413 2111 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:53:08.342448 2111 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:53:08.343603 2111 hybrid_clock.cc:648] HybridClock initialized: now 1790704388343579 us; error 38 us; skew 500 ppm
Sep 29 17:53:08 dist-test-slave-53nh krb5kdc[2098](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704388, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:53:08.346269 2111 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 17:53:08.347131 2111 webserver.cc:460] Webserver started at http://127.0.98.62:45583/ using document root <none> and password file <none>
I20260929 17:53:08.347643 2111 fs_manager.cc:359] Metadata directory not provided
I20260929 17:53:08.347705 2111 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:53:08.347913 2111 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:53:08.349974 2111 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704375975685-392-0/minicluster-data/master-0/data/instance:
uuid: "72f917d01d69473394db09e8e813365f"
format_stamp: "Formatted at 2026-09-29 17:53:08 on dist-test-slave-53nh"
server_key: "52d4318c7ca82d644f7db4427ea9b69d"
server_key_iv: "f47e60d5a0df9a386d372cb23138dcd6"
server_key_version: "encryptionkey@0"
I20260929 17:53:08.350484 2111 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704375975685-392-0/minicluster-data/master-0/wal/instance:
uuid: "72f917d01d69473394db09e8e813365f"
format_stamp: "Formatted at 2026-09-29 17:53:08 on dist-test-slave-53nh"
server_key: "52d4318c7ca82d644f7db4427ea9b69d"
server_key_iv: "f47e60d5a0df9a386d372cb23138dcd6"
server_key_version: "encryptionkey@0"
I20260929 17:53:08.354048 2111 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.001s
I20260929 17:53:08.356380 2126 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:53:08.357668 2111 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 17:53:08.357800 2111 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704375975685-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704375975685-392-0/minicluster-data/master-0/wal
uuid: "72f917d01d69473394db09e8e813365f"
format_stamp: "Formatted at 2026-09-29 17:53:08 on dist-test-slave-53nh"
server_key: "52d4318c7ca82d644f7db4427ea9b69d"
server_key_iv: "f47e60d5a0df9a386d372cb23138dcd6"
server_key_version: "encryptionkey@0"
I20260929 17:53:08.357904 2111 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704375975685-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704375975685-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704375975685-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:53:08.379947 2111 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:53:08.382887 2111 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:53:08.383066 2111 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:53:08.390647 2111 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42197
I20260929 17:53:08.390657 2188 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42197 every 8 connection(s)
I20260929 17:53:08.391566 2111 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704375975685-392-0/minicluster-data/master-0/data/info.pb
I20260929 17:53:08.394297 2189 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:53:08.399647 2189 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 72f917d01d69473394db09e8e813365f: Bootstrap starting.
I20260929 17:53:08.399739 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskup3Soe/build/debug/bin/kudu as pid 2111
I20260929 17:53:08.399832 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704375975685-392-0/minicluster-data/master-0/wal/instance
I20260929 17:53:08.400097 392 external_mini_cluster.cc:1428] Setting key 78fe1ba65682074e65579e6854839cb7
I20260929 17:53:08.402200 2189 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 72f917d01d69473394db09e8e813365f: Neither blocks nor log segments found. Creating new log.
I20260929 17:53:08.402889 2189 log.cc:824] T 00000000000000000000000000000000 P 72f917d01d69473394db09e8e813365f: Log is configured to *not* fsync() on all Append() calls
I20260929 17:53:08.404603 2189 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 72f917d01d69473394db09e8e813365f: No bootstrap required, opened a new log
Sep 29 17:53:08 dist-test-slave-53nh krb5kdc[2098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704388, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 17:53:08.408159 2189 raft_consensus.cc:348] T 00000000000000000000000000000000 P 72f917d01d69473394db09e8e813365f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72f917d01d69473394db09e8e813365f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42197 } }
I20260929 17:53:08.408463 2189 raft_consensus.cc:374] T 00000000000000000000000000000000 P 72f917d01d69473394db09e8e813365f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:53:08.408519 2189 raft_consensus.cc:729] T 00000000000000000000000000000000 P 72f917d01d69473394db09e8e813365f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72f917d01d69473394db09e8e813365f, State: Initialized, Role: FOLLOWER
I20260929 17:53:08.409029 2189 consensus_queue.cc:260] T 00000000000000000000000000000000 P 72f917d01d69473394db09e8e813365f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72f917d01d69473394db09e8e813365f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42197 } }
I20260929 17:53:08.409174 2189 raft_consensus.cc:388] T 00000000000000000000000000000000 P 72f917d01d69473394db09e8e813365f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 17:53:08.409238 2189 raft_consensus.cc:482] T 00000000000000000000000000000000 P 72f917d01d69473394db09e8e813365f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 17:53:08.409341 2189 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 72f917d01d69473394db09e8e813365f [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:53:08.410504 2189 raft_consensus.cc:504] T 00000000000000000000000000000000 P 72f917d01d69473394db09e8e813365f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72f917d01d69473394db09e8e813365f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42197 } }
I20260929 17:53:08.410902 2189 leader_election.cc:302] T 00000000000000000000000000000000 P 72f917d01d69473394db09e8e813365f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 72f917d01d69473394db09e8e813365f; no voters:
I20260929 17:53:08.411234 2189 leader_election.cc:288] T 00000000000000000000000000000000 P 72f917d01d69473394db09e8e813365f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 17:53:08.411363 2194 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 72f917d01d69473394db09e8e813365f [term 1 FOLLOWER]: Leader election won for term 1
I20260929 17:53:08.411561 2194 raft_consensus.cc:686] T 00000000000000000000000000000000 P 72f917d01d69473394db09e8e813365f [term 1 LEADER]: Becoming Leader. State: Replica: 72f917d01d69473394db09e8e813365f, State: Running, Role: LEADER
I20260929 17:53:08.411911 2194 consensus_queue.cc:237] T 00000000000000000000000000000000 P 72f917d01d69473394db09e8e813365f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72f917d01d69473394db09e8e813365f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42197 } }
I20260929 17:53:08.412582 2189 sys_catalog.cc:564] T 00000000000000000000000000000000 P 72f917d01d69473394db09e8e813365f [sys.catalog]: configured and running, proceeding with master startup.
I20260929 17:53:08.413056 2194 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72f917d01d69473394db09e8e813365f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 72f917d01d69473394db09e8e813365f. Latest consensus state: current_term: 1 leader_uuid: "72f917d01d69473394db09e8e813365f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72f917d01d69473394db09e8e813365f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42197 } } }
I20260929 17:53:08.413669 2196 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72f917d01d69473394db09e8e813365f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "72f917d01d69473394db09e8e813365f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72f917d01d69473394db09e8e813365f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42197 } } }
I20260929 17:53:08.413785 2194 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72f917d01d69473394db09e8e813365f [sys.catalog]: This master's current role is: LEADER
I20260929 17:53:08.413836 2196 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72f917d01d69473394db09e8e813365f [sys.catalog]: This master's current role is: LEADER
I20260929 17:53:08.414341 2192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:08.401381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33102 (local address 127.0.98.62:42197)
0929 17:53:08.401789 (+ 408us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:08.401805 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:08.401840 (+ 35us) server_negotiation.cc:407] Connection header received
0929 17:53:08.402505 (+ 665us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:08.402524 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:08.402854 (+ 330us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:08.403166 (+ 312us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:53:08.403924 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:08.404908 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:08.405743 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:08.406203 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:08.406348 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:08.408926 (+ 2578us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:53:08.408931 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:53:08.408944 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:53:08.408971 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:53:08.410860 (+ 1889us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:53:08.411407 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:53:08.411414 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:53:08.411421 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:53:08.411563 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:53:08.411868 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:53:08.411872 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:53:08.411874 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:53:08.412381 (+ 507us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:53:08.412563 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0929 17:53:08.412944 (+ 381us) server_negotiation.cc:299] Negotiation successful
0929 17:53:08.413264 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272,"thread_start_us":118,"threads_started":1}
W20260929 17:53:08.417444 2209 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 72f917d01d69473394db09e8e813365f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 17:53:08.417507 2209 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 17:53:08.417577 2203 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 17:53:08.418246 2203 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 17:53:08.424332 2203 catalog_manager.cc:1347] Generated new cluster ID: 1d6a54212ff04e60a1ae1aeb4c4c6956
I20260929 17:53:08.424400 2203 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 17:53:08.440582 2203 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 17:53:08.441627 2203 catalog_manager.cc:1504] Loading token signing keys...
I20260929 17:53:08.449381 2203 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 72f917d01d69473394db09e8e813365f: Generated new TSK 0
I20260929 17:53:08.450048 2203 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704375975685-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704375975685-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260929 17:53:08.481769 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskup3Soe/build/debug/bin/kudu
/tmp/dist-test-taskup3Soe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704375975685-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704375975685-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704375975685-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704375975685-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42197
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704375975685-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704375975685-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704375975685-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704375975685-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:53:08.589123 2215 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:53:08.589393 2215 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:53:08.589439 2215 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:53:08.593482 2215 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:53:08.593549 2215 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:53:08.593593 2215 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260929 17:53:08.593607 2215 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 17:53:08.593652 2215 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 17:53:08.598948 2215 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704375975685-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704375975685-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704375975685-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704375975685-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:42197
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2215
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704375975685-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:53:08.600103 2215 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:53:08.601233 2215 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:53:08.610882 2220 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:53:08.610949 2215 server_base.cc:1034] running on GCE node
W20260929 17:53:08.610882 2223 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:53:08.610888 2221 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:53:08.611538 2215 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:53:08.612156 2215 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:53:08.613308 2215 hybrid_clock.cc:648] HybridClock initialized: now 1790704388613286 us; error 33 us; skew 500 ppm
Sep 29 17:53:08 dist-test-slave-53nh krb5kdc[2098](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704388, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:53:08.616029 2215 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260929 17:53:08.616904 2215 webserver.cc:460] Webserver started at http://127.0.98.1:37579/ using document root <none> and password file <none>
I20260929 17:53:08.617470 2215 fs_manager.cc:359] Metadata directory not provided
I20260929 17:53:08.617537 2215 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:53:08.617731 2215 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:53:08.619735 2215 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704375975685-392-0/minicluster-data/ts-0/data/instance:
uuid: "9e91172c416f417386d7a002c121ca48"
format_stamp: "Formatted at 2026-09-29 17:53:08 on dist-test-slave-53nh"
server_key: "231b77d5da12c762395a6bf06daf0902"
server_key_iv: "f631215e6f11ac6d1a37a682c6ce48cb"
server_key_version: "encryptionkey@0"
I20260929 17:53:08.620285 2215 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704375975685-392-0/minicluster-data/ts-0/wal/instance:
uuid: "9e91172c416f417386d7a002c121ca48"
format_stamp: "Formatted at 2026-09-29 17:53:08 on dist-test-slave-53nh"
server_key: "231b77d5da12c762395a6bf06daf0902"
server_key_iv: "f631215e6f11ac6d1a37a682c6ce48cb"
server_key_version: "encryptionkey@0"
I20260929 17:53:08.623906 2215 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260929 17:53:08.626255 2230 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:53:08.627517 2215 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 17:53:08.627614 2215 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704375975685-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704375975685-392-0/minicluster-data/ts-0/wal
uuid: "9e91172c416f417386d7a002c121ca48"
format_stamp: "Formatted at 2026-09-29 17:53:08 on dist-test-slave-53nh"
server_key: "231b77d5da12c762395a6bf06daf0902"
server_key_iv: "f631215e6f11ac6d1a37a682c6ce48cb"
server_key_version: "encryptionkey@0"
I20260929 17:53:08.627722 2215 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704375975685-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704375975685-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704375975685-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:53:08.642735 2215 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:53:08.645776 2215 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:53:08.645923 2215 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:53:08.647276 2215 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:53:08.647326 2215 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:53:08.647378 2215 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:53:08.647409 2215 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 17:53:08.656589 2215 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37663
I20260929 17:53:08.657454 2345 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37663 every 8 connection(s)
Sep 29 17:53:08 dist-test-slave-53nh krb5kdc[2098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704388, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 17:53:08.658180 2215 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704375975685-392-0/minicluster-data/ts-0/data/info.pb
I20260929 17:53:08.661952 2192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:08.648208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46853 (local address 127.0.98.62:42197)
0929 17:53:08.648351 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:08.648356 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:08.649259 (+ 903us) server_negotiation.cc:407] Connection header received
0929 17:53:08.651060 (+ 1801us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:08.651064 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:08.651123 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:08.651190 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:53:08.652959 (+ 1769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:08.653794 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:08.655154 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:08.655520 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:08.655635 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:08.658351 (+ 2716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:53:08.658355 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:53:08.658359 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:53:08.658390 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:53:08.659881 (+ 1491us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:53:08.660382 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:53:08.660386 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:53:08.660388 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:53:08.660467 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:53:08.660798 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:53:08.660802 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:53:08.660803 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:53:08.661042 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:53:08.661133 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0929 17:53:08.661650 (+ 517us) server_negotiation.cc:299] Negotiation successful
0929 17:53:08.661779 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 17:53:08.662755 2239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:08.648585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42197 (local address 127.0.98.1:46853)
0929 17:53:08.649104 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:08.649157 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0929 17:53:08.650810 (+ 1653us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:53:08.651623 (+ 813us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:53:08.651632 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:53:08.652086 (+ 454us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:53:08.652795 (+ 709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:08.652814 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:08.653933 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:08.653937 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:08.655035 (+ 1098us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:08.655046 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:08.655688 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:08.655692 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:08.655893 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:08.656540 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:53:08.656559 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:53:08.658187 (+ 1628us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:53:08.659996 (+ 1809us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:53:08.660001 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:53:08.660015 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:53:08.660288 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:53:08.660550 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:53:08.660553 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:53:08.660556 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:53:08.660691 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:53:08.661135 (+ 444us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:53:08.661142 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:53:08.661407 (+ 265us) client_negotiation.cc:769] Sending connection context
0929 17:53:08.661606 (+ 199us) client_negotiation.cc:240] Negotiation successful
0929 17:53:08.661908 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":115,"threads_started":1}
I20260929 17:53:08.663926 2346 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42197
I20260929 17:53:08.664065 2346 heartbeater.cc:464] Registering TS with master...
I20260929 17:53:08.664657 2346 heartbeater.cc:511] Master 127.0.98.62:42197 requested a full tablet report, sending...
I20260929 17:53:08.666325 2142 ts_manager.cc:194] Registered new tserver with Master: 9e91172c416f417386d7a002c121ca48 (127.0.98.1:37663)
I20260929 17:53:08.667421 2142 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:46853
I20260929 17:53:08.668018 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskup3Soe/build/debug/bin/kudu as pid 2215
I20260929 17:53:08.668119 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704375975685-392-0/minicluster-data/ts-0/wal/instance
I20260929 17:53:08.668349 392 external_mini_cluster.cc:1428] Setting key 09315dfff038ed48137041da47852328
I20260929 17:53:08.672806 2352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:08.666135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42197 (local address 127.0.98.1:53471)
0929 17:53:08.666371 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:08.666383 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 17:53:08.666469 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:53:08.666673 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:53:08.666675 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:53:08.666849 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:53:08.666934 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:08.666939 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:08.667623 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:08.667626 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:08.668489 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:08.668499 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:08.668938 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:08.668942 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:08.669046 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:08.669541 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:53:08.669553 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:53:08.669853 (+ 300us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:53:08.671315 (+ 1462us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:53:08.671318 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:53:08.671320 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:53:08.671673 (+ 353us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:53:08.671990 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:53:08.671992 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:53:08.671993 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:53:08.672065 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:53:08.672413 (+ 348us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:53:08.672415 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:53:08.672482 (+ 67us) client_negotiation.cc:769] Sending connection context
0929 17:53:08.672551 (+ 69us) client_negotiation.cc:240] Negotiation successful
0929 17:53:08.672648 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"mutex_wait_us":1,"thread_start_us":69,"threads_started":1}
I20260929 17:53:08.672863 2192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:08.666222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53471 (local address 127.0.98.62:42197)
0929 17:53:08.666323 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:08.666327 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:08.666446 (+ 119us) server_negotiation.cc:407] Connection header received
0929 17:53:08.666538 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:08.666541 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:08.666579 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:08.666648 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:53:08.667030 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:08.667512 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:08.668604 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:08.668837 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:08.668914 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:08.669956 (+ 1042us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:53:08.669959 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:53:08.669962 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:53:08.669983 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:53:08.671207 (+ 1224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:53:08.671805 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:53:08.671809 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:53:08.671810 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:53:08.671903 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:53:08.672135 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:53:08.672137 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:53:08.672139 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:53:08.672339 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:53:08.672429 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0929 17:53:08.672618 (+ 189us) server_negotiation.cc:299] Negotiation successful
0929 17:53:08.672728 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704375975685-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704375975685-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260929 17:53:08.694150 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskup3Soe/build/debug/bin/kudu
/tmp/dist-test-taskup3Soe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704375975685-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704375975685-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704375975685-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704375975685-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42197
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704375975685-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704375975685-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704375975685-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704375975685-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:53:08.800614 2355 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:53:08.800891 2355 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:53:08.800937 2355 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:53:08.804988 2355 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:53:08.805083 2355 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:53:08.805123 2355 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260929 17:53:08.805136 2355 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 17:53:08.805174 2355 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 17:53:08.810485 2355 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704375975685-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704375975685-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704375975685-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704375975685-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:42197
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2355
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704375975685-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:53:08.811702 2355 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:53:08.812783 2355 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:53:08.822082 2360 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:53:08.822075 2361 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:53:08.822117 2355 server_base.cc:1034] running on GCE node
W20260929 17:53:08.822075 2363 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:53:08.822618 2355 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:53:08.823225 2355 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:53:08.824379 2355 hybrid_clock.cc:648] HybridClock initialized: now 1790704388824362 us; error 30 us; skew 500 ppm
Sep 29 17:53:08 dist-test-slave-53nh krb5kdc[2098](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704388, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:53:08.826941 2355 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260929 17:53:08.827801 2355 webserver.cc:460] Webserver started at http://127.0.98.2:38147/ using document root <none> and password file <none>
I20260929 17:53:08.828327 2355 fs_manager.cc:359] Metadata directory not provided
I20260929 17:53:08.828387 2355 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:53:08.828576 2355 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:53:08.830612 2355 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704375975685-392-0/minicluster-data/ts-1/data/instance:
uuid: "b367e800a8fb4f0aa965712d54ee86f5"
format_stamp: "Formatted at 2026-09-29 17:53:08 on dist-test-slave-53nh"
server_key: "c2e8cb32f10d777011a615cf87a2ca3a"
server_key_iv: "6e59f57ebba20a0a04c5936ebd42da7f"
server_key_version: "encryptionkey@0"
I20260929 17:53:08.831179 2355 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704375975685-392-0/minicluster-data/ts-1/wal/instance:
uuid: "b367e800a8fb4f0aa965712d54ee86f5"
format_stamp: "Formatted at 2026-09-29 17:53:08 on dist-test-slave-53nh"
server_key: "c2e8cb32f10d777011a615cf87a2ca3a"
server_key_iv: "6e59f57ebba20a0a04c5936ebd42da7f"
server_key_version: "encryptionkey@0"
I20260929 17:53:08.834800 2355 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260929 17:53:08.837152 2370 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:53:08.838445 2355 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 17:53:08.838544 2355 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704375975685-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704375975685-392-0/minicluster-data/ts-1/wal
uuid: "b367e800a8fb4f0aa965712d54ee86f5"
format_stamp: "Formatted at 2026-09-29 17:53:08 on dist-test-slave-53nh"
server_key: "c2e8cb32f10d777011a615cf87a2ca3a"
server_key_iv: "6e59f57ebba20a0a04c5936ebd42da7f"
server_key_version: "encryptionkey@0"
I20260929 17:53:08.838652 2355 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704375975685-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704375975685-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704375975685-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:53:08.850091 2355 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:53:08.852859 2355 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:53:08.853075 2355 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:53:08.854485 2355 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:53:08.854532 2355 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:53:08.854570 2355 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:53:08.854592 2355 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 17:53:08 dist-test-slave-53nh krb5kdc[2098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704388, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 17:53:08.864579 2355 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46719
I20260929 17:53:08.864634 2485 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46719 every 8 connection(s)
I20260929 17:53:08.865563 2355 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704375975685-392-0/minicluster-data/ts-1/data/info.pb
I20260929 17:53:08.866971 2192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:08.855153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54327 (local address 127.0.98.62:42197)
0929 17:53:08.855298 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:08.855303 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:08.856089 (+ 786us) server_negotiation.cc:407] Connection header received
0929 17:53:08.857073 (+ 984us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:08.857078 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:08.857132 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:08.857186 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:53:08.858542 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:08.859187 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:08.860401 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:08.860624 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:08.860719 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:08.863433 (+ 2714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:53:08.863437 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:53:08.863440 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:53:08.863467 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:53:08.864966 (+ 1499us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:53:08.865525 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:53:08.865529 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:53:08.865531 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:53:08.865618 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:53:08.865934 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:53:08.865938 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:53:08.865939 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:53:08.866152 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:53:08.866237 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0929 17:53:08.866742 (+ 505us) server_negotiation.cc:299] Negotiation successful
0929 17:53:08.866848 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 17:53:08.867718 2379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:08.855529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42197 (local address 127.0.98.2:54327)
0929 17:53:08.855943 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:08.855990 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 17:53:08.856814 (+ 824us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:53:08.857588 (+ 774us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:53:08.857596 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:53:08.857909 (+ 313us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:53:08.858406 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:08.858422 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:08.859316 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:08.859319 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:08.860287 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:08.860296 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:08.860766 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:08.860770 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:08.860934 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:08.861533 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:53:08.861556 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:53:08.863274 (+ 1718us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:53:08.865160 (+ 1886us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:53:08.865166 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:53:08.865180 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:53:08.865434 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:53:08.865690 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:53:08.865693 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:53:08.865696 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:53:08.865830 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:53:08.866282 (+ 452us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:53:08.866288 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:53:08.866517 (+ 229us) client_negotiation.cc:769] Sending connection context
0929 17:53:08.866666 (+ 149us) client_negotiation.cc:240] Negotiation successful
0929 17:53:08.866914 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":95,"threads_started":1}
I20260929 17:53:08.868858 2486 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42197
I20260929 17:53:08.869019 2486 heartbeater.cc:464] Registering TS with master...
I20260929 17:53:08.869480 2486 heartbeater.cc:511] Master 127.0.98.62:42197 requested a full tablet report, sending...
I20260929 17:53:08.869910 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskup3Soe/build/debug/bin/kudu as pid 2355
I20260929 17:53:08.870020 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704375975685-392-0/minicluster-data/ts-1/wal/instance
I20260929 17:53:08.870309 392 external_mini_cluster.cc:1428] Setting key e8c2e118db275d5a3b8c3fe5ad88e010
I20260929 17:53:08.870558 2143 ts_manager.cc:194] Registered new tserver with Master: b367e800a8fb4f0aa965712d54ee86f5 (127.0.98.2:46719)
I20260929 17:53:08.871244 2143 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:54327
I20260929 17:53:08.878383 2492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:08.870918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42197 (local address 127.0.98.2:45961)
0929 17:53:08.871153 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:08.871164 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 17:53:08.871251 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:53:08.871448 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:53:08.871451 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:53:08.871643 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:53:08.871736 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:08.871742 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:08.872987 (+ 1245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:08.872990 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:08.873673 (+ 683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:08.873682 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:08.874098 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:08.874102 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:08.874320 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:08.874810 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:53:08.874821 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:53:08.875104 (+ 283us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:53:08.876900 (+ 1796us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:53:08.876903 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:53:08.876904 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:53:08.877128 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:53:08.877417 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:53:08.877419 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:53:08.877421 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:53:08.877508 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:53:08.877988 (+ 480us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:53:08.877991 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:53:08.878067 (+ 76us) client_negotiation.cc:769] Sending connection context
0929 17:53:08.878148 (+ 81us) client_negotiation.cc:240] Negotiation successful
0929 17:53:08.878247 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":88,"threads_started":1}
I20260929 17:53:08.878494 2192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:08.870886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45961 (local address 127.0.98.62:42197)
0929 17:53:08.870986 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:08.870989 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:08.871227 (+ 238us) server_negotiation.cc:407] Connection header received
0929 17:53:08.871320 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:08.871323 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:08.871365 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:08.871426 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:53:08.872101 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:08.872834 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:08.873778 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:08.874003 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:08.874087 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:08.875255 (+ 1168us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:53:08.875259 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:53:08.875262 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:53:08.875287 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:53:08.876789 (+ 1502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:53:08.877213 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:53:08.877217 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:53:08.877218 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:53:08.877306 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:53:08.877599 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:53:08.877602 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:53:08.877604 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:53:08.877883 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:53:08.877989 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0929 17:53:08.878224 (+ 235us) server_negotiation.cc:299] Negotiation successful
0929 17:53:08.878343 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704375975685-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704375975685-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260929 17:53:08.896937 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskup3Soe/build/debug/bin/kudu
/tmp/dist-test-taskup3Soe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704375975685-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704375975685-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704375975685-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704375975685-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42197
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704375975685-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704375975685-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704375975685-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704375975685-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:53:09.005494 2495 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:53:09.005777 2495 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:53:09.005826 2495 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:53:09.009905 2495 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:53:09.009965 2495 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:53:09.010005 2495 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260929 17:53:09.010031 2495 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 17:53:09.010076 2495 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 17:53:09.015653 2495 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704375975685-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704375975685-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704375975685-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704375975685-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:42197
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2495
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704375975685-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:53:09.016747 2495 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:53:09.017827 2495 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:53:09.028252 2503 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:53:09.028245 2501 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:53:09.028247 2500 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:53:09.028594 2495 server_base.cc:1034] running on GCE node
I20260929 17:53:09.028985 2495 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:53:09.029641 2495 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:53:09.030822 2495 hybrid_clock.cc:648] HybridClock initialized: now 1790704389030789 us; error 45 us; skew 500 ppm
Sep 29 17:53:09 dist-test-slave-53nh krb5kdc[2098](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704389, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:53:09.033553 2495 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260929 17:53:09.034380 2495 webserver.cc:460] Webserver started at http://127.0.98.3:43285/ using document root <none> and password file <none>
I20260929 17:53:09.034899 2495 fs_manager.cc:359] Metadata directory not provided
I20260929 17:53:09.034962 2495 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:53:09.035152 2495 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:53:09.037202 2495 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704375975685-392-0/minicluster-data/ts-2/data/instance:
uuid: "3cddbb0910424ec2b410a25791b1ba46"
format_stamp: "Formatted at 2026-09-29 17:53:09 on dist-test-slave-53nh"
server_key: "e4908d6434830f615f16d1c2d5693785"
server_key_iv: "b2c4c5f601199868ee1c869d7ab56feb"
server_key_version: "encryptionkey@0"
I20260929 17:53:09.037751 2495 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704375975685-392-0/minicluster-data/ts-2/wal/instance:
uuid: "3cddbb0910424ec2b410a25791b1ba46"
format_stamp: "Formatted at 2026-09-29 17:53:09 on dist-test-slave-53nh"
server_key: "e4908d6434830f615f16d1c2d5693785"
server_key_iv: "b2c4c5f601199868ee1c869d7ab56feb"
server_key_version: "encryptionkey@0"
I20260929 17:53:09.041369 2495 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260929 17:53:09.043800 2510 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:53:09.044972 2495 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 17:53:09.045100 2495 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704375975685-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704375975685-392-0/minicluster-data/ts-2/wal
uuid: "3cddbb0910424ec2b410a25791b1ba46"
format_stamp: "Formatted at 2026-09-29 17:53:09 on dist-test-slave-53nh"
server_key: "e4908d6434830f615f16d1c2d5693785"
server_key_iv: "b2c4c5f601199868ee1c869d7ab56feb"
server_key_version: "encryptionkey@0"
I20260929 17:53:09.045248 2495 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704375975685-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704375975685-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704375975685-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:53:09.063643 2495 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:53:09.066677 2495 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:53:09.066830 2495 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:53:09.068172 2495 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:53:09.068228 2495 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:53:09.068272 2495 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:53:09.068300 2495 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 17:53:09 dist-test-slave-53nh krb5kdc[2098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704389, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 17:53:09.078169 2495 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37811
I20260929 17:53:09.078214 2625 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37811 every 8 connection(s)
I20260929 17:53:09.079201 2495 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704375975685-392-0/minicluster-data/ts-2/data/info.pb
I20260929 17:53:09.080986 2192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:09.069035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42039 (local address 127.0.98.62:42197)
0929 17:53:09.069149 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:09.069153 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:09.070217 (+ 1064us) server_negotiation.cc:407] Connection header received
0929 17:53:09.071332 (+ 1115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:09.071336 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:09.071393 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:09.071498 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:53:09.073086 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:09.073707 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:09.074874 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:09.075088 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:09.075176 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:09.077497 (+ 2321us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:53:09.077500 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:53:09.077503 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:53:09.077531 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:53:09.079143 (+ 1612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:53:09.079639 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:53:09.079643 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:53:09.079645 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:53:09.079747 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:53:09.080103 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:53:09.080107 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:53:09.080109 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:53:09.080291 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:53:09.080366 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0929 17:53:09.080778 (+ 412us) server_negotiation.cc:299] Negotiation successful
0929 17:53:09.080870 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260929 17:53:09.081717 2519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:09.069611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42197 (local address 127.0.98.3:42039)
0929 17:53:09.070086 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:09.070127 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 17:53:09.071061 (+ 934us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:53:09.071729 (+ 668us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:53:09.071741 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:53:09.072208 (+ 467us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:53:09.072903 (+ 695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:09.072926 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:09.073829 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:09.073833 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:09.074762 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:09.074771 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:09.075231 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:09.075235 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:09.075390 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:09.075904 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:53:09.075919 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:53:09.077334 (+ 1415us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:53:09.079247 (+ 1913us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:53:09.079253 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:53:09.079267 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:53:09.079518 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:53:09.079842 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:53:09.079845 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:53:09.079847 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:53:09.080003 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:53:09.080371 (+ 368us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:53:09.080375 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:53:09.080582 (+ 207us) client_negotiation.cc:769] Sending connection context
0929 17:53:09.080722 (+ 140us) client_negotiation.cc:240] Negotiation successful
0929 17:53:09.080960 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":122,"threads_started":1}
I20260929 17:53:09.082772 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskup3Soe/build/debug/bin/kudu as pid 2495
I20260929 17:53:09.082863 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704375975685-392-0/minicluster-data/ts-2/wal/instance
I20260929 17:53:09.083094 392 external_mini_cluster.cc:1428] Setting key cebaa74e1ea9254b753cfbe8ff431daf
I20260929 17:53:09.084746 2626 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42197
I20260929 17:53:09.084872 2626 heartbeater.cc:464] Registering TS with master...
I20260929 17:53:09.085330 2626 heartbeater.cc:511] Master 127.0.98.62:42197 requested a full tablet report, sending...
I20260929 17:53:09.086179 2143 ts_manager.cc:194] Registered new tserver with Master: 3cddbb0910424ec2b410a25791b1ba46 (127.0.98.3:37811)
I20260929 17:53:09.086812 2143 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:42039
I20260929 17:53:09.091599 2192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:09.084960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53839 (local address 127.0.98.62:42197)
0929 17:53:09.085084 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:09.085088 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:09.085942 (+ 854us) server_negotiation.cc:407] Connection header received
0929 17:53:09.086024 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:09.086026 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:09.086062 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:09.086120 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:53:09.086566 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:09.087161 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:09.087993 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:09.088296 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:09.088395 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:09.089358 (+ 963us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:53:09.089361 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:53:09.089363 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:53:09.089383 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:53:09.090462 (+ 1079us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:53:09.090795 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:53:09.090798 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:53:09.090799 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:53:09.090860 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:53:09.091039 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:53:09.091040 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:53:09.091042 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:53:09.091174 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:53:09.091240 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0929 17:53:09.091414 (+ 174us) server_negotiation.cc:299] Negotiation successful
0929 17:53:09.091493 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":18,"server-negotiator.queue_time_us":58}
I20260929 17:53:09.091606 2632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:09.084955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42197 (local address 127.0.98.3:53839)
0929 17:53:09.085864 (+ 909us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:09.085878 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 17:53:09.085956 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:53:09.086153 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:53:09.086156 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:53:09.086359 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:53:09.086470 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:09.086477 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:09.087287 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:09.087290 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:09.087882 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:09.087891 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:09.088391 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:09.088393 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:09.088456 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:09.088889 (+ 433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:53:09.088900 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:53:09.089245 (+ 345us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:53:09.090551 (+ 1306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:53:09.090554 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:53:09.090555 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:53:09.090724 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:53:09.090919 (+ 195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:53:09.090921 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:53:09.090922 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:53:09.090983 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:53:09.091232 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:53:09.091234 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:53:09.091304 (+ 70us) client_negotiation.cc:769] Sending connection context
0929 17:53:09.091373 (+ 69us) client_negotiation.cc:240] Negotiation successful
0929 17:53:09.091466 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":833,"mutex_wait_us":1,"thread_start_us":110,"threads_started":1}
I20260929 17:53:09.096455 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 17:53:09 dist-test-slave-53nh krb5kdc[2098](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704389, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 17:53:09 dist-test-slave-53nh krb5kdc[2098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704389, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 17:53:09.119455 2192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:09.111931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57472 (local address 127.0.98.62:42197)
0929 17:53:09.112052 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:09.112056 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:09.112067 (+ 11us) server_negotiation.cc:407] Connection header received
0929 17:53:09.112112 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:09.112114 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:09.112153 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:09.112222 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:53:09.112912 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:09.113594 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:09.114406 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:09.114623 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:09.114709 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:09.116924 (+ 2215us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:53:09.116928 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:53:09.116930 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:53:09.116954 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:53:09.118183 (+ 1229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:53:09.118531 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:53:09.118533 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:53:09.118534 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:53:09.118598 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:53:09.118772 (+ 174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:53:09.118774 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:53:09.118775 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:53:09.118929 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:53:09.119020 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0929 17:53:09.119172 (+ 152us) server_negotiation.cc:299] Negotiation successful
0929 17:53:09.119291 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 17:53:09.122282 2143 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57472:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 17:53:09.125178 2143 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 17:53:09.138895 2645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:09.133528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37811 (local address 127.0.0.1:43038)
0929 17:53:09.134529 (+ 1001us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:09.134543 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 17:53:09.134634 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:53:09.135061 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:53:09.135064 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:53:09.135080 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:53:09.135174 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:09.135180 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:09.136354 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:09.136358 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:09.137462 (+ 1104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:09.137470 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:09.138408 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:09.138412 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:09.138508 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:09.138531 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 17:53:09.138606 (+ 75us) client_negotiation.cc:240] Negotiation successful
0929 17:53:09.138754 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":948,"mutex_wait_us":1,"thread_start_us":110,"threads_started":1}
I20260929 17:53:09.138904 2647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:09.133932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43038 (local address 127.0.98.3:37811)
0929 17:53:09.134406 (+ 474us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:09.134418 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:09.134636 (+ 218us) server_negotiation.cc:407] Connection header received
0929 17:53:09.134734 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:09.134739 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:09.134946 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:09.135081 (+ 135us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:53:09.135269 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:09.136239 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:09.137572 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:09.138303 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:09.138397 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:09.138550 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0929 17:53:09.138645 (+ 95us) server_negotiation.cc:299] Negotiation successful
0929 17:53:09.138747 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":115,"threads_started":1}
I20260929 17:53:09.138980 2644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:09.133304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59114 (local address 127.0.98.1:37663)
0929 17:53:09.133717 (+ 413us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:09.133731 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:09.133756 (+ 25us) server_negotiation.cc:407] Connection header received
0929 17:53:09.133837 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:09.133843 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:09.134063 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:09.134221 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:53:09.134694 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:09.135894 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:09.137417 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:09.138033 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:09.138141 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:09.138302 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0929 17:53:09.138734 (+ 432us) server_negotiation.cc:299] Negotiation successful
0929 17:53:09.138831 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":93,"threads_started":1}
I20260929 17:53:09.139424 2643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:09.133179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37663 (local address 127.0.0.1:59114)
0929 17:53:09.133493 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:09.133539 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 17:53:09.133666 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:53:09.134399 (+ 733us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:53:09.134405 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:53:09.134430 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:53:09.134560 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:09.134568 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:09.135985 (+ 1417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:09.135988 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:09.137289 (+ 1301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:09.137299 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:09.138362 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:09.138366 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:09.138471 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:09.138508 (+ 37us) client_negotiation.cc:769] Sending connection context
0929 17:53:09.138618 (+ 110us) client_negotiation.cc:240] Negotiation successful
0929 17:53:09.139287 (+ 669us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"mutex_wait_us":563,"thread_start_us":74,"threads_started":1}
I20260929 17:53:09.139423 2646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:09.133904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46719 (local address 127.0.0.1:54000)
0929 17:53:09.134767 (+ 863us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:09.134780 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 17:53:09.134856 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:53:09.135435 (+ 579us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:53:09.135438 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:53:09.135452 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:53:09.135524 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:09.135530 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:09.136762 (+ 1232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:09.136765 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:09.137842 (+ 1077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:09.137851 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:09.139034 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:09.139037 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:09.139120 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:09.139142 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 17:53:09.139267 (+ 125us) client_negotiation.cc:240] Negotiation successful
0929 17:53:09.139321 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":806,"thread_start_us":87,"threads_started":1}
I20260929 17:53:09.139555 2648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:09.134375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54000 (local address 127.0.98.2:46719)
0929 17:53:09.134868 (+ 493us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:09.134878 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:09.134898 (+ 20us) server_negotiation.cc:407] Connection header received
0929 17:53:09.134964 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:09.134969 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:09.135158 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:09.135295 (+ 137us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:53:09.135641 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:09.136624 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:09.137991 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:09.138748 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:09.138836 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:09.138972 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0929 17:53:09.139331 (+ 359us) server_negotiation.cc:299] Negotiation successful
0929 17:53:09.139420 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":402,"thread_start_us":92,"threads_started":1}
I20260929 17:53:09.142293 2420 tablet_service.cc:1467] Processing CreateTablet for tablet f68fd58dffbf4d02bd8b5e8769f9f723 (DEFAULT_TABLE table=test-table [id=0592a8beae4743e9bb0123a939e794ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 17:53:09.142329 2560 tablet_service.cc:1467] Processing CreateTablet for tablet f68fd58dffbf4d02bd8b5e8769f9f723 (DEFAULT_TABLE table=test-table [id=0592a8beae4743e9bb0123a939e794ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 17:53:09.142329 2280 tablet_service.cc:1467] Processing CreateTablet for tablet f68fd58dffbf4d02bd8b5e8769f9f723 (DEFAULT_TABLE table=test-table [id=0592a8beae4743e9bb0123a939e794ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 17:53:09.143316 2420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f68fd58dffbf4d02bd8b5e8769f9f723. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:53:09.143333 2560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f68fd58dffbf4d02bd8b5e8769f9f723. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:53:09.143326 2280 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f68fd58dffbf4d02bd8b5e8769f9f723. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:53:09.150597 2649 tablet_bootstrap.cc:492] T f68fd58dffbf4d02bd8b5e8769f9f723 P b367e800a8fb4f0aa965712d54ee86f5: Bootstrap starting.
I20260929 17:53:09.151849 2651 tablet_bootstrap.cc:492] T f68fd58dffbf4d02bd8b5e8769f9f723 P 3cddbb0910424ec2b410a25791b1ba46: Bootstrap starting.
I20260929 17:53:09.152878 2649 tablet_bootstrap.cc:654] T f68fd58dffbf4d02bd8b5e8769f9f723 P b367e800a8fb4f0aa965712d54ee86f5: Neither blocks nor log segments found. Creating new log.
I20260929 17:53:09.153364 2650 tablet_bootstrap.cc:492] T f68fd58dffbf4d02bd8b5e8769f9f723 P 9e91172c416f417386d7a002c121ca48: Bootstrap starting.
I20260929 17:53:09.153625 2651 tablet_bootstrap.cc:654] T f68fd58dffbf4d02bd8b5e8769f9f723 P 3cddbb0910424ec2b410a25791b1ba46: Neither blocks nor log segments found. Creating new log.
I20260929 17:53:09.153807 2649 log.cc:824] T f68fd58dffbf4d02bd8b5e8769f9f723 P b367e800a8fb4f0aa965712d54ee86f5: Log is configured to *not* fsync() on all Append() calls
I20260929 17:53:09.154363 2651 log.cc:824] T f68fd58dffbf4d02bd8b5e8769f9f723 P 3cddbb0910424ec2b410a25791b1ba46: Log is configured to *not* fsync() on all Append() calls
I20260929 17:53:09.155393 2650 tablet_bootstrap.cc:654] T f68fd58dffbf4d02bd8b5e8769f9f723 P 9e91172c416f417386d7a002c121ca48: Neither blocks nor log segments found. Creating new log.
I20260929 17:53:09.156055 2649 tablet_bootstrap.cc:492] T f68fd58dffbf4d02bd8b5e8769f9f723 P b367e800a8fb4f0aa965712d54ee86f5: No bootstrap required, opened a new log
I20260929 17:53:09.156190 2651 tablet_bootstrap.cc:492] T f68fd58dffbf4d02bd8b5e8769f9f723 P 3cddbb0910424ec2b410a25791b1ba46: No bootstrap required, opened a new log
I20260929 17:53:09.156262 2649 ts_tablet_manager.cc:1397] T f68fd58dffbf4d02bd8b5e8769f9f723 P b367e800a8fb4f0aa965712d54ee86f5: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.003s
I20260929 17:53:09.156263 2650 log.cc:824] T f68fd58dffbf4d02bd8b5e8769f9f723 P 9e91172c416f417386d7a002c121ca48: Log is configured to *not* fsync() on all Append() calls
I20260929 17:53:09.156368 2651 ts_tablet_manager.cc:1397] T f68fd58dffbf4d02bd8b5e8769f9f723 P 3cddbb0910424ec2b410a25791b1ba46: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260929 17:53:09.158231 2650 tablet_bootstrap.cc:492] T f68fd58dffbf4d02bd8b5e8769f9f723 P 9e91172c416f417386d7a002c121ca48: No bootstrap required, opened a new log
I20260929 17:53:09.158416 2650 ts_tablet_manager.cc:1397] T f68fd58dffbf4d02bd8b5e8769f9f723 P 9e91172c416f417386d7a002c121ca48: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260929 17:53:09.159405 2651 raft_consensus.cc:348] T f68fd58dffbf4d02bd8b5e8769f9f723 P 3cddbb0910424ec2b410a25791b1ba46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e91172c416f417386d7a002c121ca48" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37663 } } peers { permanent_uuid: "b367e800a8fb4f0aa965712d54ee86f5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46719 } } peers { permanent_uuid: "3cddbb0910424ec2b410a25791b1ba46" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37811 } }
I20260929 17:53:09.159624 2651 raft_consensus.cc:374] T f68fd58dffbf4d02bd8b5e8769f9f723 P 3cddbb0910424ec2b410a25791b1ba46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:53:09.159694 2651 raft_consensus.cc:729] T f68fd58dffbf4d02bd8b5e8769f9f723 P 3cddbb0910424ec2b410a25791b1ba46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3cddbb0910424ec2b410a25791b1ba46, State: Initialized, Role: FOLLOWER
I20260929 17:53:09.160111 2651 consensus_queue.cc:260] T f68fd58dffbf4d02bd8b5e8769f9f723 P 3cddbb0910424ec2b410a25791b1ba46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e91172c416f417386d7a002c121ca48" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37663 } } peers { permanent_uuid: "b367e800a8fb4f0aa965712d54ee86f5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46719 } } peers { permanent_uuid: "3cddbb0910424ec2b410a25791b1ba46" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37811 } }
I20260929 17:53:09.160058 2649 raft_consensus.cc:348] T f68fd58dffbf4d02bd8b5e8769f9f723 P b367e800a8fb4f0aa965712d54ee86f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e91172c416f417386d7a002c121ca48" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37663 } } peers { permanent_uuid: "b367e800a8fb4f0aa965712d54ee86f5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46719 } } peers { permanent_uuid: "3cddbb0910424ec2b410a25791b1ba46" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37811 } }
I20260929 17:53:09.160341 2649 raft_consensus.cc:374] T f68fd58dffbf4d02bd8b5e8769f9f723 P b367e800a8fb4f0aa965712d54ee86f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:53:09.160410 2649 raft_consensus.cc:729] T f68fd58dffbf4d02bd8b5e8769f9f723 P b367e800a8fb4f0aa965712d54ee86f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b367e800a8fb4f0aa965712d54ee86f5, State: Initialized, Role: FOLLOWER
I20260929 17:53:09.160858 2649 consensus_queue.cc:260] T f68fd58dffbf4d02bd8b5e8769f9f723 P b367e800a8fb4f0aa965712d54ee86f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e91172c416f417386d7a002c121ca48" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37663 } } peers { permanent_uuid: "b367e800a8fb4f0aa965712d54ee86f5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46719 } } peers { permanent_uuid: "3cddbb0910424ec2b410a25791b1ba46" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37811 } }
I20260929 17:53:09.161090 2651 ts_tablet_manager.cc:1428] T f68fd58dffbf4d02bd8b5e8769f9f723 P 3cddbb0910424ec2b410a25791b1ba46: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260929 17:53:09.161571 2486 heartbeater.cc:503] Master 127.0.98.62:42197 was elected leader, sending a full tablet report...
I20260929 17:53:09.161607 2626 heartbeater.cc:503] Master 127.0.98.62:42197 was elected leader, sending a full tablet report...
I20260929 17:53:09.162232 2650 raft_consensus.cc:348] T f68fd58dffbf4d02bd8b5e8769f9f723 P 9e91172c416f417386d7a002c121ca48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e91172c416f417386d7a002c121ca48" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37663 } } peers { permanent_uuid: "b367e800a8fb4f0aa965712d54ee86f5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46719 } } peers { permanent_uuid: "3cddbb0910424ec2b410a25791b1ba46" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37811 } }
I20260929 17:53:09.162477 2650 raft_consensus.cc:374] T f68fd58dffbf4d02bd8b5e8769f9f723 P 9e91172c416f417386d7a002c121ca48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:53:09.162526 2650 raft_consensus.cc:729] T f68fd58dffbf4d02bd8b5e8769f9f723 P 9e91172c416f417386d7a002c121ca48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e91172c416f417386d7a002c121ca48, State: Initialized, Role: FOLLOWER
I20260929 17:53:09.162966 2650 consensus_queue.cc:260] T f68fd58dffbf4d02bd8b5e8769f9f723 P 9e91172c416f417386d7a002c121ca48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e91172c416f417386d7a002c121ca48" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37663 } } peers { permanent_uuid: "b367e800a8fb4f0aa965712d54ee86f5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46719 } } peers { permanent_uuid: "3cddbb0910424ec2b410a25791b1ba46" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37811 } }
I20260929 17:53:09.163435 2649 ts_tablet_manager.cc:1428] T f68fd58dffbf4d02bd8b5e8769f9f723 P b367e800a8fb4f0aa965712d54ee86f5: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260929 17:53:09.163619 2346 heartbeater.cc:503] Master 127.0.98.62:42197 was elected leader, sending a full tablet report...
I20260929 17:53:09.163925 2650 ts_tablet_manager.cc:1428] T f68fd58dffbf4d02bd8b5e8769f9f723 P 9e91172c416f417386d7a002c121ca48: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260929 17:53:09.213203 2657 raft_consensus.cc:482] T f68fd58dffbf4d02bd8b5e8769f9f723 P 9e91172c416f417386d7a002c121ca48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 17:53:09.213439 2657 raft_consensus.cc:504] T f68fd58dffbf4d02bd8b5e8769f9f723 P 9e91172c416f417386d7a002c121ca48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e91172c416f417386d7a002c121ca48" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37663 } } peers { permanent_uuid: "b367e800a8fb4f0aa965712d54ee86f5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46719 } } peers { permanent_uuid: "3cddbb0910424ec2b410a25791b1ba46" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37811 } }
I20260929 17:53:09.214669 2657 leader_election.cc:288] T f68fd58dffbf4d02bd8b5e8769f9f723 P 9e91172c416f417386d7a002c121ca48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b367e800a8fb4f0aa965712d54ee86f5 (127.0.98.2:46719), 3cddbb0910424ec2b410a25791b1ba46 (127.0.98.3:37811)
I20260929 17:53:09.219676 2648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:09.214999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57697 (local address 127.0.98.2:46719)
0929 17:53:09.215120 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:09.215125 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:09.215306 (+ 181us) server_negotiation.cc:407] Connection header received
0929 17:53:09.215435 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:09.215437 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:09.215485 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:09.215564 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:53:09.215936 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:09.216748 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:09.218444 (+ 1696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:09.219036 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:09.219146 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:09.219209 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0929 17:53:09.219477 (+ 268us) server_negotiation.cc:299] Negotiation successful
0929 17:53:09.219549 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 17:53:09.219676 2647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:09.215023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46215 (local address 127.0.98.3:37811)
0929 17:53:09.215224 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:09.215228 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:09.215345 (+ 117us) server_negotiation.cc:407] Connection header received
0929 17:53:09.215430 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:09.215433 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:09.215485 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:09.215564 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:53:09.215934 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:09.216748 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:09.218434 (+ 1686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:09.219058 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:09.219146 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:09.219209 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0929 17:53:09.219473 (+ 264us) server_negotiation.cc:299] Negotiation successful
0929 17:53:09.219549 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115}
I20260929 17:53:09.219740 2658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:09.214903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37811 (local address 127.0.98.1:46215)
0929 17:53:09.215271 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:09.215282 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 17:53:09.215351 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:53:09.215581 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:53:09.215584 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:53:09.215622 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:53:09.215828 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:09.215834 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:09.216856 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:09.216859 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:09.218311 (+ 1452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:09.218323 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:09.219169 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:09.219173 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:09.219327 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:09.219369 (+ 42us) client_negotiation.cc:769] Sending connection context
0929 17:53:09.219445 (+ 76us) client_negotiation.cc:240] Negotiation successful
0929 17:53:09.219540 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"mutex_wait_us":229,"spinlock_wait_cycles":4608,"thread_start_us":58,"threads_started":1}
I20260929 17:53:09.219743 2659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:09.214902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46719 (local address 127.0.98.1:57697)
0929 17:53:09.215204 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:09.215219 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 17:53:09.215342 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:53:09.215581 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:53:09.215584 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:53:09.215620 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:53:09.215821 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:09.215828 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:09.216856 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:09.216859 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:09.218342 (+ 1483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:09.218348 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:09.219169 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:09.219173 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:09.219333 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:09.219369 (+ 36us) client_negotiation.cc:769] Sending connection context
0929 17:53:09.219445 (+ 76us) client_negotiation.cc:240] Negotiation successful
0929 17:53:09.219540 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"mutex_wait_us":194,"spinlock_wait_cycles":27776,"thread_start_us":97,"threads_started":1}
I20260929 17:53:09.220526 2580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f68fd58dffbf4d02bd8b5e8769f9f723" candidate_uuid: "9e91172c416f417386d7a002c121ca48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3cddbb0910424ec2b410a25791b1ba46" is_pre_election: true
I20260929 17:53:09.220526 2440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f68fd58dffbf4d02bd8b5e8769f9f723" candidate_uuid: "9e91172c416f417386d7a002c121ca48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b367e800a8fb4f0aa965712d54ee86f5" is_pre_election: true
I20260929 17:53:09.220860 2440 raft_consensus.cc:2454] T f68fd58dffbf4d02bd8b5e8769f9f723 P b367e800a8fb4f0aa965712d54ee86f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9e91172c416f417386d7a002c121ca48 in term 0.
I20260929 17:53:09.220860 2580 raft_consensus.cc:2454] T f68fd58dffbf4d02bd8b5e8769f9f723 P 3cddbb0910424ec2b410a25791b1ba46 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9e91172c416f417386d7a002c121ca48 in term 0.
I20260929 17:53:09.221446 2232 leader_election.cc:302] T f68fd58dffbf4d02bd8b5e8769f9f723 P 9e91172c416f417386d7a002c121ca48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3cddbb0910424ec2b410a25791b1ba46, 9e91172c416f417386d7a002c121ca48; no voters:
I20260929 17:53:09.221704 2657 raft_consensus.cc:2781] T f68fd58dffbf4d02bd8b5e8769f9f723 P 9e91172c416f417386d7a002c121ca48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 17:53:09.221788 2657 raft_consensus.cc:482] T f68fd58dffbf4d02bd8b5e8769f9f723 P 9e91172c416f417386d7a002c121ca48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 17:53:09.221837 2657 raft_consensus.cc:3037] T f68fd58dffbf4d02bd8b5e8769f9f723 P 9e91172c416f417386d7a002c121ca48 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:53:09.222867 2657 raft_consensus.cc:504] T f68fd58dffbf4d02bd8b5e8769f9f723 P 9e91172c416f417386d7a002c121ca48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e91172c416f417386d7a002c121ca48" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37663 } } peers { permanent_uuid: "b367e800a8fb4f0aa965712d54ee86f5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46719 } } peers { permanent_uuid: "3cddbb0910424ec2b410a25791b1ba46" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37811 } }
I20260929 17:53:09.223186 2657 leader_election.cc:288] T f68fd58dffbf4d02bd8b5e8769f9f723 P 9e91172c416f417386d7a002c121ca48 [CANDIDATE]: Term 1 election: Requested vote from peers b367e800a8fb4f0aa965712d54ee86f5 (127.0.98.2:46719), 3cddbb0910424ec2b410a25791b1ba46 (127.0.98.3:37811)
I20260929 17:53:09.223572 2580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f68fd58dffbf4d02bd8b5e8769f9f723" candidate_uuid: "9e91172c416f417386d7a002c121ca48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3cddbb0910424ec2b410a25791b1ba46"
I20260929 17:53:09.223740 2580 raft_consensus.cc:3037] T f68fd58dffbf4d02bd8b5e8769f9f723 P 3cddbb0910424ec2b410a25791b1ba46 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:53:09.224896 2580 raft_consensus.cc:2454] T f68fd58dffbf4d02bd8b5e8769f9f723 P 3cddbb0910424ec2b410a25791b1ba46 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e91172c416f417386d7a002c121ca48 in term 1.
I20260929 17:53:09.225204 2440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f68fd58dffbf4d02bd8b5e8769f9f723" candidate_uuid: "9e91172c416f417386d7a002c121ca48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b367e800a8fb4f0aa965712d54ee86f5"
I20260929 17:53:09.225231 2232 leader_election.cc:302] T f68fd58dffbf4d02bd8b5e8769f9f723 P 9e91172c416f417386d7a002c121ca48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3cddbb0910424ec2b410a25791b1ba46, 9e91172c416f417386d7a002c121ca48; no voters:
I20260929 17:53:09.225337 2440 raft_consensus.cc:3037] T f68fd58dffbf4d02bd8b5e8769f9f723 P b367e800a8fb4f0aa965712d54ee86f5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:53:09.225436 2657 raft_consensus.cc:2781] T f68fd58dffbf4d02bd8b5e8769f9f723 P 9e91172c416f417386d7a002c121ca48 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 17:53:09.225596 2657 raft_consensus.cc:686] T f68fd58dffbf4d02bd8b5e8769f9f723 P 9e91172c416f417386d7a002c121ca48 [term 1 LEADER]: Becoming Leader. State: Replica: 9e91172c416f417386d7a002c121ca48, State: Running, Role: LEADER
I20260929 17:53:09.225936 2657 consensus_queue.cc:237] T f68fd58dffbf4d02bd8b5e8769f9f723 P 9e91172c416f417386d7a002c121ca48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e91172c416f417386d7a002c121ca48" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37663 } } peers { permanent_uuid: "b367e800a8fb4f0aa965712d54ee86f5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46719 } } peers { permanent_uuid: "3cddbb0910424ec2b410a25791b1ba46" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37811 } }
I20260929 17:53:09.226804 2440 raft_consensus.cc:2454] T f68fd58dffbf4d02bd8b5e8769f9f723 P b367e800a8fb4f0aa965712d54ee86f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e91172c416f417386d7a002c121ca48 in term 1.
I20260929 17:53:09.229074 2143 catalog_manager.cc:5495] T f68fd58dffbf4d02bd8b5e8769f9f723 P 9e91172c416f417386d7a002c121ca48 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9e91172c416f417386d7a002c121ca48 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "9e91172c416f417386d7a002c121ca48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e91172c416f417386d7a002c121ca48" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37663 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b367e800a8fb4f0aa965712d54ee86f5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46719 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3cddbb0910424ec2b410a25791b1ba46" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37811 } health_report { overall_health: UNKNOWN } } }
Sep 29 17:53:09 dist-test-slave-53nh krb5kdc[2098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704388, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 17:53:09.278249 2667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:09.270697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42197 (local address 127.0.0.1:57478)
0929 17:53:09.271008 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:09.271024 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 17:53:09.271130 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:53:09.271375 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:53:09.271377 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:53:09.271765 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:53:09.271882 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:09.271889 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:09.272616 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:09.272619 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:09.273188 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:09.273195 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:09.273591 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:09.273593 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:09.273648 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:09.274208 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:53:09.274252 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:53:09.275594 (+ 1342us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:53:09.276951 (+ 1357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:53:09.276959 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:53:09.276965 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:53:09.277193 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:53:09.277426 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:53:09.277429 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:53:09.277430 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:53:09.277498 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:53:09.277769 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:53:09.277774 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:53:09.277871 (+ 97us) client_negotiation.cc:769] Sending connection context
0929 17:53:09.277953 (+ 82us) client_negotiation.cc:240] Negotiation successful
0929 17:53:09.278068 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":116,"threads_started":1}
I20260929 17:53:09.278254 2192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:09.270772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57478 (local address 127.0.98.62:42197)
0929 17:53:09.270912 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:09.270915 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:09.271095 (+ 180us) server_negotiation.cc:407] Connection header received
0929 17:53:09.271237 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:09.271240 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:09.271284 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:09.271349 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:53:09.271993 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:09.272520 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:09.273292 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:09.273500 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:09.273577 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:09.275711 (+ 2134us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:53:09.275715 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:53:09.275717 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:53:09.275736 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:53:09.276861 (+ 1125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:53:09.277294 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:53:09.277296 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:53:09.277298 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:53:09.277363 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:53:09.277555 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:53:09.277557 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:53:09.277558 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:53:09.277701 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:53:09.277781 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0929 17:53:09.277994 (+ 213us) server_negotiation.cc:299] Negotiation successful
0929 17:53:09.278123 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":32,"server-negotiator.queue_time_us":62}
I20260929 17:53:09.282119 2143 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57478:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260929 17:53:09.282871 2143 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 17:53:09.287518 2280 tablet_service.cc:1467] Processing CreateTablet for tablet 7aae06a3cca74641bcc1ec3fe9586518 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=8faa290fcc1e4b31a67bb8d9bd762dec]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 17:53:09.287559 2420 tablet_service.cc:1467] Processing CreateTablet for tablet 7aae06a3cca74641bcc1ec3fe9586518 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=8faa290fcc1e4b31a67bb8d9bd762dec]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 17:53:09.287858 2560 tablet_service.cc:1467] Processing CreateTablet for tablet 7aae06a3cca74641bcc1ec3fe9586518 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=8faa290fcc1e4b31a67bb8d9bd762dec]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 17:53:09.288249 2280 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7aae06a3cca74641bcc1ec3fe9586518. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:53:09.288259 2420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7aae06a3cca74641bcc1ec3fe9586518. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:53:09.288367 2560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7aae06a3cca74641bcc1ec3fe9586518. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:53:09.290604 2650 tablet_bootstrap.cc:492] T 7aae06a3cca74641bcc1ec3fe9586518 P 9e91172c416f417386d7a002c121ca48: Bootstrap starting.
I20260929 17:53:09.291018 2649 tablet_bootstrap.cc:492] T 7aae06a3cca74641bcc1ec3fe9586518 P b367e800a8fb4f0aa965712d54ee86f5: Bootstrap starting.
I20260929 17:53:09.291177 2651 tablet_bootstrap.cc:492] T 7aae06a3cca74641bcc1ec3fe9586518 P 3cddbb0910424ec2b410a25791b1ba46: Bootstrap starting.
I20260929 17:53:09.291787 2650 tablet_bootstrap.cc:654] T 7aae06a3cca74641bcc1ec3fe9586518 P 9e91172c416f417386d7a002c121ca48: Neither blocks nor log segments found. Creating new log.
I20260929 17:53:09.291944 2649 tablet_bootstrap.cc:654] T 7aae06a3cca74641bcc1ec3fe9586518 P b367e800a8fb4f0aa965712d54ee86f5: Neither blocks nor log segments found. Creating new log.
I20260929 17:53:09.292054 2651 tablet_bootstrap.cc:654] T 7aae06a3cca74641bcc1ec3fe9586518 P 3cddbb0910424ec2b410a25791b1ba46: Neither blocks nor log segments found. Creating new log.
I20260929 17:53:09.293015 2650 tablet_bootstrap.cc:492] T 7aae06a3cca74641bcc1ec3fe9586518 P 9e91172c416f417386d7a002c121ca48: No bootstrap required, opened a new log
I20260929 17:53:09.293102 2650 ts_tablet_manager.cc:1397] T 7aae06a3cca74641bcc1ec3fe9586518 P 9e91172c416f417386d7a002c121ca48: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260929 17:53:09.293355 2649 tablet_bootstrap.cc:492] T 7aae06a3cca74641bcc1ec3fe9586518 P b367e800a8fb4f0aa965712d54ee86f5: No bootstrap required, opened a new log
I20260929 17:53:09.293426 2649 ts_tablet_manager.cc:1397] T 7aae06a3cca74641bcc1ec3fe9586518 P b367e800a8fb4f0aa965712d54ee86f5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20260929 17:53:09.293481 2651 tablet_bootstrap.cc:492] T 7aae06a3cca74641bcc1ec3fe9586518 P 3cddbb0910424ec2b410a25791b1ba46: No bootstrap required, opened a new log
I20260929 17:53:09.293537 2651 ts_tablet_manager.cc:1397] T 7aae06a3cca74641bcc1ec3fe9586518 P 3cddbb0910424ec2b410a25791b1ba46: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 17:53:09.293560 2650 raft_consensus.cc:348] T 7aae06a3cca74641bcc1ec3fe9586518 P 9e91172c416f417386d7a002c121ca48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e91172c416f417386d7a002c121ca48" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37663 } } peers { permanent_uuid: "b367e800a8fb4f0aa965712d54ee86f5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46719 } } peers { permanent_uuid: "3cddbb0910424ec2b410a25791b1ba46" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37811 } }
I20260929 17:53:09.293663 2650 raft_consensus.cc:374] T 7aae06a3cca74641bcc1ec3fe9586518 P 9e91172c416f417386d7a002c121ca48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:53:09.293694 2650 raft_consensus.cc:729] T 7aae06a3cca74641bcc1ec3fe9586518 P 9e91172c416f417386d7a002c121ca48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e91172c416f417386d7a002c121ca48, State: Initialized, Role: FOLLOWER
I20260929 17:53:09.293813 2650 consensus_queue.cc:260] T 7aae06a3cca74641bcc1ec3fe9586518 P 9e91172c416f417386d7a002c121ca48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e91172c416f417386d7a002c121ca48" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37663 } } peers { permanent_uuid: "b367e800a8fb4f0aa965712d54ee86f5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46719 } } peers { permanent_uuid: "3cddbb0910424ec2b410a25791b1ba46" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37811 } }
I20260929 17:53:09.293841 2649 raft_consensus.cc:348] T 7aae06a3cca74641bcc1ec3fe9586518 P b367e800a8fb4f0aa965712d54ee86f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e91172c416f417386d7a002c121ca48" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37663 } } peers { permanent_uuid: "b367e800a8fb4f0aa965712d54ee86f5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46719 } } peers { permanent_uuid: "3cddbb0910424ec2b410a25791b1ba46" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37811 } }
I20260929 17:53:09.293927 2649 raft_consensus.cc:374] T 7aae06a3cca74641bcc1ec3fe9586518 P b367e800a8fb4f0aa965712d54ee86f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:53:09.293956 2649 raft_consensus.cc:729] T 7aae06a3cca74641bcc1ec3fe9586518 P b367e800a8fb4f0aa965712d54ee86f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b367e800a8fb4f0aa965712d54ee86f5, State: Initialized, Role: FOLLOWER
I20260929 17:53:09.293896 2651 raft_consensus.cc:348] T 7aae06a3cca74641bcc1ec3fe9586518 P 3cddbb0910424ec2b410a25791b1ba46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e91172c416f417386d7a002c121ca48" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37663 } } peers { permanent_uuid: "b367e800a8fb4f0aa965712d54ee86f5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46719 } } peers { permanent_uuid: "3cddbb0910424ec2b410a25791b1ba46" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37811 } }
I20260929 17:53:09.293993 2651 raft_consensus.cc:374] T 7aae06a3cca74641bcc1ec3fe9586518 P 3cddbb0910424ec2b410a25791b1ba46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:53:09.294025 2651 raft_consensus.cc:729] T 7aae06a3cca74641bcc1ec3fe9586518 P 3cddbb0910424ec2b410a25791b1ba46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3cddbb0910424ec2b410a25791b1ba46, State: Initialized, Role: FOLLOWER
I20260929 17:53:09.294061 2649 consensus_queue.cc:260] T 7aae06a3cca74641bcc1ec3fe9586518 P b367e800a8fb4f0aa965712d54ee86f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e91172c416f417386d7a002c121ca48" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37663 } } peers { permanent_uuid: "b367e800a8fb4f0aa965712d54ee86f5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46719 } } peers { permanent_uuid: "3cddbb0910424ec2b410a25791b1ba46" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37811 } }
I20260929 17:53:09.294081 2657 tablet_replica.cc:439] T 7aae06a3cca74641bcc1ec3fe9586518 P 9e91172c416f417386d7a002c121ca48: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e91172c416f417386d7a002c121ca48" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37663 } } peers { permanent_uuid: "b367e800a8fb4f0aa965712d54ee86f5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46719 } } peers { permanent_uuid: "3cddbb0910424ec2b410a25791b1ba46" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37811 } } }
I20260929 17:53:09.294134 2651 consensus_queue.cc:260] T 7aae06a3cca74641bcc1ec3fe9586518 P 3cddbb0910424ec2b410a25791b1ba46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e91172c416f417386d7a002c121ca48" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37663 } } peers { permanent_uuid: "b367e800a8fb4f0aa965712d54ee86f5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46719 } } peers { permanent_uuid: "3cddbb0910424ec2b410a25791b1ba46" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37811 } }
I20260929 17:53:09.294207 2657 tablet_replica.cc:442] T 7aae06a3cca74641bcc1ec3fe9586518 P 9e91172c416f417386d7a002c121ca48: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 17:53:09.294292 2649 ts_tablet_manager.cc:1428] T 7aae06a3cca74641bcc1ec3fe9586518 P b367e800a8fb4f0aa965712d54ee86f5: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260929 17:53:09.294382 2651 ts_tablet_manager.cc:1428] T 7aae06a3cca74641bcc1ec3fe9586518 P 3cddbb0910424ec2b410a25791b1ba46: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 17:53:09.294560 2656 tablet_replica.cc:439] T 7aae06a3cca74641bcc1ec3fe9586518 P b367e800a8fb4f0aa965712d54ee86f5: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e91172c416f417386d7a002c121ca48" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37663 } } peers { permanent_uuid: "b367e800a8fb4f0aa965712d54ee86f5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46719 } } peers { permanent_uuid: "3cddbb0910424ec2b410a25791b1ba46" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37811 } } }
I20260929 17:53:09.294615 2655 tablet_replica.cc:439] T 7aae06a3cca74641bcc1ec3fe9586518 P 3cddbb0910424ec2b410a25791b1ba46: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e91172c416f417386d7a002c121ca48" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37663 } } peers { permanent_uuid: "b367e800a8fb4f0aa965712d54ee86f5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46719 } } peers { permanent_uuid: "3cddbb0910424ec2b410a25791b1ba46" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37811 } } }
I20260929 17:53:09.294687 2656 tablet_replica.cc:442] T 7aae06a3cca74641bcc1ec3fe9586518 P b367e800a8fb4f0aa965712d54ee86f5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 17:53:09.294747 2655 tablet_replica.cc:442] T 7aae06a3cca74641bcc1ec3fe9586518 P 3cddbb0910424ec2b410a25791b1ba46: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 17:53:09.295588 2650 ts_tablet_manager.cc:1428] T 7aae06a3cca74641bcc1ec3fe9586518 P 9e91172c416f417386d7a002c121ca48: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
W20260929 17:53:09.330374 2627 tablet.cc:2369] T 7aae06a3cca74641bcc1ec3fe9586518 P 3cddbb0910424ec2b410a25791b1ba46: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 17:53:09.366640 2487 tablet.cc:2369] T 7aae06a3cca74641bcc1ec3fe9586518 P b367e800a8fb4f0aa965712d54ee86f5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 17:53:09.409408 2347 tablet.cc:2369] T 7aae06a3cca74641bcc1ec3fe9586518 P 9e91172c416f417386d7a002c121ca48: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 17:53:09.424662 2656 raft_consensus.cc:482] T 7aae06a3cca74641bcc1ec3fe9586518 P b367e800a8fb4f0aa965712d54ee86f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 17:53:09.424805 2656 raft_consensus.cc:504] T 7aae06a3cca74641bcc1ec3fe9586518 P b367e800a8fb4f0aa965712d54ee86f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e91172c416f417386d7a002c121ca48" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37663 } } peers { permanent_uuid: "b367e800a8fb4f0aa965712d54ee86f5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46719 } } peers { permanent_uuid: "3cddbb0910424ec2b410a25791b1ba46" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37811 } }
I20260929 17:53:09.426177 2656 leader_election.cc:288] T 7aae06a3cca74641bcc1ec3fe9586518 P b367e800a8fb4f0aa965712d54ee86f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9e91172c416f417386d7a002c121ca48 (127.0.98.1:37663), 3cddbb0910424ec2b410a25791b1ba46 (127.0.98.3:37811)
I20260929 17:53:09.430392 2644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:09.426747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33887 (local address 127.0.98.1:37663)
0929 17:53:09.427168 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:09.427172 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:09.427188 (+ 16us) server_negotiation.cc:407] Connection header received
0929 17:53:09.427239 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:09.427242 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:09.427302 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:09.427373 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:53:09.427634 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:09.428370 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:09.429484 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:09.429889 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:09.429979 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:09.430040 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0929 17:53:09.430208 (+ 168us) server_negotiation.cc:299] Negotiation successful
0929 17:53:09.430278 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260929 17:53:09.430410 2671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:09.426399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37663 (local address 127.0.98.2:33887)
0929 17:53:09.426924 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:09.426940 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 17:53:09.427053 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:53:09.427416 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:53:09.427419 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:53:09.427431 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:53:09.427539 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:09.427544 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:09.428481 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:09.428484 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:09.429379 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:09.429388 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:09.429992 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:09.429995 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:09.430106 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:09.430126 (+ 20us) client_negotiation.cc:769] Sending connection context
0929 17:53:09.430187 (+ 61us) client_negotiation.cc:240] Negotiation successful
0929 17:53:09.430256 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":446,"mutex_wait_us":20,"thread_start_us":126,"threads_started":1}
I20260929 17:53:09.430495 2647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:09.426691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56573 (local address 127.0.98.3:37811)
0929 17:53:09.426983 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:09.426988 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:09.427290 (+ 302us) server_negotiation.cc:407] Connection header received
0929 17:53:09.427396 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:09.427433 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:09.427507 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:09.427608 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:53:09.427854 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:09.428569 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:09.429632 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:09.430016 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:09.430103 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:09.430159 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0929 17:53:09.430317 (+ 158us) server_negotiation.cc:299] Negotiation successful
0929 17:53:09.430380 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260929 17:53:09.430689 2672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:09.426415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37811 (local address 127.0.98.2:56573)
0929 17:53:09.427224 (+ 809us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:09.427235 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 17:53:09.427311 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:53:09.427650 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:53:09.427652 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:53:09.427668 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:53:09.427744 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:09.427749 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:09.428684 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:09.428687 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:09.429523 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:09.429532 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:09.430117 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:09.430120 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:09.430214 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:09.430235 (+ 21us) client_negotiation.cc:769] Sending connection context
0929 17:53:09.430297 (+ 62us) client_negotiation.cc:240] Negotiation successful
0929 17:53:09.430360 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":752,"mutex_wait_us":1,"thread_start_us":107,"threads_started":1}
I20260929 17:53:09.431103 2580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7aae06a3cca74641bcc1ec3fe9586518" candidate_uuid: "b367e800a8fb4f0aa965712d54ee86f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3cddbb0910424ec2b410a25791b1ba46" is_pre_election: true
I20260929 17:53:09.431196 2300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7aae06a3cca74641bcc1ec3fe9586518" candidate_uuid: "b367e800a8fb4f0aa965712d54ee86f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e91172c416f417386d7a002c121ca48" is_pre_election: true
I20260929 17:53:09.431260 2580 raft_consensus.cc:2454] T 7aae06a3cca74641bcc1ec3fe9586518 P 3cddbb0910424ec2b410a25791b1ba46 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b367e800a8fb4f0aa965712d54ee86f5 in term 0.
I20260929 17:53:09.431414 2300 raft_consensus.cc:2454] T 7aae06a3cca74641bcc1ec3fe9586518 P 9e91172c416f417386d7a002c121ca48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b367e800a8fb4f0aa965712d54ee86f5 in term 0.
I20260929 17:53:09.431583 2372 leader_election.cc:302] T 7aae06a3cca74641bcc1ec3fe9586518 P b367e800a8fb4f0aa965712d54ee86f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3cddbb0910424ec2b410a25791b1ba46, b367e800a8fb4f0aa965712d54ee86f5; no voters:
I20260929 17:53:09.431861 2656 raft_consensus.cc:2781] T 7aae06a3cca74641bcc1ec3fe9586518 P b367e800a8fb4f0aa965712d54ee86f5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 17:53:09.431919 2656 raft_consensus.cc:482] T 7aae06a3cca74641bcc1ec3fe9586518 P b367e800a8fb4f0aa965712d54ee86f5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 17:53:09.431964 2656 raft_consensus.cc:3037] T 7aae06a3cca74641bcc1ec3fe9586518 P b367e800a8fb4f0aa965712d54ee86f5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:53:09.432873 2656 raft_consensus.cc:504] T 7aae06a3cca74641bcc1ec3fe9586518 P b367e800a8fb4f0aa965712d54ee86f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e91172c416f417386d7a002c121ca48" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37663 } } peers { permanent_uuid: "b367e800a8fb4f0aa965712d54ee86f5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46719 } } peers { permanent_uuid: "3cddbb0910424ec2b410a25791b1ba46" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37811 } }
I20260929 17:53:09.433205 2656 leader_election.cc:288] T 7aae06a3cca74641bcc1ec3fe9586518 P b367e800a8fb4f0aa965712d54ee86f5 [CANDIDATE]: Term 1 election: Requested vote from peers 9e91172c416f417386d7a002c121ca48 (127.0.98.1:37663), 3cddbb0910424ec2b410a25791b1ba46 (127.0.98.3:37811)
I20260929 17:53:09.433529 2300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7aae06a3cca74641bcc1ec3fe9586518" candidate_uuid: "b367e800a8fb4f0aa965712d54ee86f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e91172c416f417386d7a002c121ca48"
I20260929 17:53:09.433537 2580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7aae06a3cca74641bcc1ec3fe9586518" candidate_uuid: "b367e800a8fb4f0aa965712d54ee86f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3cddbb0910424ec2b410a25791b1ba46"
I20260929 17:53:09.433652 2580 raft_consensus.cc:3037] T 7aae06a3cca74641bcc1ec3fe9586518 P 3cddbb0910424ec2b410a25791b1ba46 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:53:09.433653 2300 raft_consensus.cc:3037] T 7aae06a3cca74641bcc1ec3fe9586518 P 9e91172c416f417386d7a002c121ca48 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:53:09.434718 2300 raft_consensus.cc:2454] T 7aae06a3cca74641bcc1ec3fe9586518 P 9e91172c416f417386d7a002c121ca48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b367e800a8fb4f0aa965712d54ee86f5 in term 1.
I20260929 17:53:09.434726 2580 raft_consensus.cc:2454] T 7aae06a3cca74641bcc1ec3fe9586518 P 3cddbb0910424ec2b410a25791b1ba46 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b367e800a8fb4f0aa965712d54ee86f5 in term 1.
I20260929 17:53:09.434979 2372 leader_election.cc:302] T 7aae06a3cca74641bcc1ec3fe9586518 P b367e800a8fb4f0aa965712d54ee86f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 3cddbb0910424ec2b410a25791b1ba46, 9e91172c416f417386d7a002c121ca48, b367e800a8fb4f0aa965712d54ee86f5; no voters:
I20260929 17:53:09.435138 2656 raft_consensus.cc:2781] T 7aae06a3cca74641bcc1ec3fe9586518 P b367e800a8fb4f0aa965712d54ee86f5 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 17:53:09.435309 2656 raft_consensus.cc:686] T 7aae06a3cca74641bcc1ec3fe9586518 P b367e800a8fb4f0aa965712d54ee86f5 [term 1 LEADER]: Becoming Leader. State: Replica: b367e800a8fb4f0aa965712d54ee86f5, State: Running, Role: LEADER
I20260929 17:53:09.435642 2656 consensus_queue.cc:237] T 7aae06a3cca74641bcc1ec3fe9586518 P b367e800a8fb4f0aa965712d54ee86f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e91172c416f417386d7a002c121ca48" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37663 } } peers { permanent_uuid: "b367e800a8fb4f0aa965712d54ee86f5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46719 } } peers { permanent_uuid: "3cddbb0910424ec2b410a25791b1ba46" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37811 } }
I20260929 17:53:09.437840 2673 tablet_replica.cc:439] T 7aae06a3cca74641bcc1ec3fe9586518 P b367e800a8fb4f0aa965712d54ee86f5: TxnStatusTablet state changed. Reason: New leader b367e800a8fb4f0aa965712d54ee86f5. Latest consensus state: current_term: 1 leader_uuid: "b367e800a8fb4f0aa965712d54ee86f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e91172c416f417386d7a002c121ca48" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37663 } } peers { permanent_uuid: "b367e800a8fb4f0aa965712d54ee86f5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46719 } } peers { permanent_uuid: "3cddbb0910424ec2b410a25791b1ba46" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37811 } } }
I20260929 17:53:09.438012 2673 tablet_replica.cc:442] T 7aae06a3cca74641bcc1ec3fe9586518 P b367e800a8fb4f0aa965712d54ee86f5: This TxnStatusTablet replica's current role is: LEADER
I20260929 17:53:09.438321 2675 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260929 17:53:09.438535 2675 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260929 17:53:09.438854 2142 catalog_manager.cc:5495] T 7aae06a3cca74641bcc1ec3fe9586518 P b367e800a8fb4f0aa965712d54ee86f5 reported cstate change: term changed from 0 to 1, leader changed from <none> to b367e800a8fb4f0aa965712d54ee86f5 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "b367e800a8fb4f0aa965712d54ee86f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e91172c416f417386d7a002c121ca48" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37663 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b367e800a8fb4f0aa965712d54ee86f5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46719 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3cddbb0910424ec2b410a25791b1ba46" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37811 } health_report { overall_health: UNKNOWN } } }
I20260929 17:53:09.453117 2667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:09.449282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46719 (local address 127.0.0.1:54012)
0929 17:53:09.449431 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:09.449446 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 17:53:09.449528 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:53:09.449846 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:53:09.449848 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:53:09.449855 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 17:53:09.449941 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:09.449945 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:09.450932 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:09.450934 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:09.451525 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:09.451533 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:09.452064 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:09.452067 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:09.452132 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:09.452172 (+ 40us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 17:53:09.452859 (+ 687us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 17:53:09.452866 (+ 7us) client_negotiation.cc:769] Sending connection context
0929 17:53:09.452904 (+ 38us) client_negotiation.cc:240] Negotiation successful
0929 17:53:09.452969 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260929 17:53:09.453135 2648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:09.449347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54012 (local address 127.0.98.2:46719)
0929 17:53:09.449471 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:09.449475 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:09.449490 (+ 15us) server_negotiation.cc:407] Connection header received
0929 17:53:09.449644 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:09.449669 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:09.449729 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:09.449834 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 17:53:09.450027 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:09.450812 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:09.451623 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:09.451968 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:09.452064 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:09.452288 (+ 224us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 17:53:09.452780 (+ 492us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 17:53:09.452885 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0929 17:53:09.452916 (+ 31us) server_negotiation.cc:299] Negotiation successful
0929 17:53:09.452969 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 17:53:09.458297 2300 raft_consensus.cc:1261] T 7aae06a3cca74641bcc1ec3fe9586518 P 9e91172c416f417386d7a002c121ca48 [term 1 FOLLOWER]: Refusing update from remote peer b367e800a8fb4f0aa965712d54ee86f5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 17:53:09.458302 2580 raft_consensus.cc:1261] T 7aae06a3cca74641bcc1ec3fe9586518 P 3cddbb0910424ec2b410a25791b1ba46 [term 1 FOLLOWER]: Refusing update from remote peer b367e800a8fb4f0aa965712d54ee86f5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 17:53:09.458930 2673 consensus_queue.cc:1035] T 7aae06a3cca74641bcc1ec3fe9586518 P b367e800a8fb4f0aa965712d54ee86f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e91172c416f417386d7a002c121ca48" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37663 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 17:53:09.459088 2656 consensus_queue.cc:1035] T 7aae06a3cca74641bcc1ec3fe9586518 P b367e800a8fb4f0aa965712d54ee86f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3cddbb0910424ec2b410a25791b1ba46" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37811 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 17:53:09.462023 2657 tablet_replica.cc:439] T 7aae06a3cca74641bcc1ec3fe9586518 P 9e91172c416f417386d7a002c121ca48: TxnStatusTablet state changed. Reason: New leader b367e800a8fb4f0aa965712d54ee86f5. Latest consensus state: current_term: 1 leader_uuid: "b367e800a8fb4f0aa965712d54ee86f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e91172c416f417386d7a002c121ca48" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37663 } } peers { permanent_uuid: "b367e800a8fb4f0aa965712d54ee86f5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46719 } } peers { permanent_uuid: "3cddbb0910424ec2b410a25791b1ba46" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37811 } } }
I20260929 17:53:09.462136 2657 tablet_replica.cc:442] T 7aae06a3cca74641bcc1ec3fe9586518 P 9e91172c416f417386d7a002c121ca48: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 17:53:09.463007 2655 tablet_replica.cc:439] T 7aae06a3cca74641bcc1ec3fe9586518 P 3cddbb0910424ec2b410a25791b1ba46: TxnStatusTablet state changed. Reason: New leader b367e800a8fb4f0aa965712d54ee86f5. Latest consensus state: current_term: 1 leader_uuid: "b367e800a8fb4f0aa965712d54ee86f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e91172c416f417386d7a002c121ca48" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37663 } } peers { permanent_uuid: "b367e800a8fb4f0aa965712d54ee86f5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46719 } } peers { permanent_uuid: "3cddbb0910424ec2b410a25791b1ba46" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37811 } } }
I20260929 17:53:09.463142 2655 tablet_replica.cc:442] T 7aae06a3cca74641bcc1ec3fe9586518 P 3cddbb0910424ec2b410a25791b1ba46: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 17:53:09.466172 2673 tablet_replica.cc:439] T 7aae06a3cca74641bcc1ec3fe9586518 P b367e800a8fb4f0aa965712d54ee86f5: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b367e800a8fb4f0aa965712d54ee86f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e91172c416f417386d7a002c121ca48" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37663 } } peers { permanent_uuid: "b367e800a8fb4f0aa965712d54ee86f5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46719 } } peers { permanent_uuid: "3cddbb0910424ec2b410a25791b1ba46" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37811 } } }
I20260929 17:53:09.466300 2673 tablet_replica.cc:442] T 7aae06a3cca74641bcc1ec3fe9586518 P b367e800a8fb4f0aa965712d54ee86f5: This TxnStatusTablet replica's current role is: LEADER
I20260929 17:53:09.466682 2656 tablet_replica.cc:439] T 7aae06a3cca74641bcc1ec3fe9586518 P b367e800a8fb4f0aa965712d54ee86f5: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b367e800a8fb4f0aa965712d54ee86f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e91172c416f417386d7a002c121ca48" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37663 } } peers { permanent_uuid: "b367e800a8fb4f0aa965712d54ee86f5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46719 } } peers { permanent_uuid: "3cddbb0910424ec2b410a25791b1ba46" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37811 } } }
I20260929 17:53:09.466776 2656 tablet_replica.cc:442] T 7aae06a3cca74641bcc1ec3fe9586518 P b367e800a8fb4f0aa965712d54ee86f5: This TxnStatusTablet replica's current role is: LEADER
I20260929 17:53:09.467175 2657 tablet_replica.cc:439] T 7aae06a3cca74641bcc1ec3fe9586518 P 9e91172c416f417386d7a002c121ca48: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b367e800a8fb4f0aa965712d54ee86f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e91172c416f417386d7a002c121ca48" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37663 } } peers { permanent_uuid: "b367e800a8fb4f0aa965712d54ee86f5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46719 } } peers { permanent_uuid: "3cddbb0910424ec2b410a25791b1ba46" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37811 } } }
I20260929 17:53:09.467275 2657 tablet_replica.cc:442] T 7aae06a3cca74641bcc1ec3fe9586518 P 9e91172c416f417386d7a002c121ca48: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 17:53:09.467628 2679 mvcc.cc:205] Tried to move back new op lower bound from 7334725179214876672 to 7334725179131060224. Current Snapshot: MvccSnapshot[applied={T|T < 7334725179214876672}]
I20260929 17:53:09.467901 2655 tablet_replica.cc:439] T 7aae06a3cca74641bcc1ec3fe9586518 P 3cddbb0910424ec2b410a25791b1ba46: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b367e800a8fb4f0aa965712d54ee86f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e91172c416f417386d7a002c121ca48" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37663 } } peers { permanent_uuid: "b367e800a8fb4f0aa965712d54ee86f5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46719 } } peers { permanent_uuid: "3cddbb0910424ec2b410a25791b1ba46" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37811 } } }
I20260929 17:53:09.467998 2655 tablet_replica.cc:442] T 7aae06a3cca74641bcc1ec3fe9586518 P 3cddbb0910424ec2b410a25791b1ba46: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 17:53:09.470356 2680 mvcc.cc:205] Tried to move back new op lower bound from 7334725179214876672 to 7334725179131060224. Current Snapshot: MvccSnapshot[applied={T|T < 7334725179214876672}]
I20260929 17:53:09.477203 2644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:09.473375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59118 (local address 127.0.98.1:37663)
0929 17:53:09.473491 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:09.473495 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:09.473555 (+ 60us) server_negotiation.cc:407] Connection header received
0929 17:53:09.473685 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:09.473688 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:09.473753 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:09.473853 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 17:53:09.474036 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:09.474834 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:09.475612 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:09.475973 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:09.476066 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:09.476351 (+ 285us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 17:53:09.476854 (+ 503us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 17:53:09.476961 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0929 17:53:09.476991 (+ 30us) server_negotiation.cc:299] Negotiation successful
0929 17:53:09.477071 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260929 17:53:09.487742 2648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:09.484292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42455 (local address 127.0.98.2:46719)
0929 17:53:09.484420 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:09.484424 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:09.484531 (+ 107us) server_negotiation.cc:407] Connection header received
0929 17:53:09.484650 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:09.484652 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:09.484694 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:09.484752 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 17:53:09.485028 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:09.485556 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:09.486783 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:09.486991 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:09.487067 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:09.487393 (+ 326us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 17:53:09.487459 (+ 66us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 17:53:09.487503 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0929 17:53:09.487588 (+ 85us) server_negotiation.cc:299] Negotiation successful
0929 17:53:09.487630 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 17:53:09.487835 2690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:09.484146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46719 (local address 127.0.98.1:42455)
0929 17:53:09.484455 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:09.484470 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 17:53:09.484571 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:53:09.484776 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:53:09.484779 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:53:09.484786 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 17:53:09.484907 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:09.484912 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:09.485646 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:09.485649 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:09.486672 (+ 1023us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:09.486683 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:09.487087 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:09.487090 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:09.487182 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:09.487232 (+ 50us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 17:53:09.487515 (+ 283us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 17:53:09.487521 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 17:53:09.487582 (+ 61us) client_negotiation.cc:240] Negotiation successful
0929 17:53:09.487658 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":104,"threads_started":1}
I20260929 17:53:09.494520 2580 raft_consensus.cc:1261] T f68fd58dffbf4d02bd8b5e8769f9f723 P 3cddbb0910424ec2b410a25791b1ba46 [term 1 FOLLOWER]: Refusing update from remote peer 9e91172c416f417386d7a002c121ca48: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 17:53:09.494860 2440 raft_consensus.cc:1261] T f68fd58dffbf4d02bd8b5e8769f9f723 P b367e800a8fb4f0aa965712d54ee86f5 [term 1 FOLLOWER]: Refusing update from remote peer 9e91172c416f417386d7a002c121ca48: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 17:53:09.494941 2660 consensus_queue.cc:1035] T f68fd58dffbf4d02bd8b5e8769f9f723 P 9e91172c416f417386d7a002c121ca48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3cddbb0910424ec2b410a25791b1ba46" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37811 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 17:53:09.495327 2660 consensus_queue.cc:1035] T f68fd58dffbf4d02bd8b5e8769f9f723 P 9e91172c416f417386d7a002c121ca48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b367e800a8fb4f0aa965712d54ee86f5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 17:53:09.504711 2678 mvcc.cc:205] Tried to move back new op lower bound from 7334725179365429248 to 7334725178272022528. Current Snapshot: MvccSnapshot[applied={T|T < 7334725179365429248}]
I20260929 17:53:09.517628 2644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:09.513982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44911 (local address 127.0.98.1:37663)
0929 17:53:09.514088 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:09.514092 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:09.514330 (+ 238us) server_negotiation.cc:407] Connection header received
0929 17:53:09.514450 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:09.514453 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:09.514501 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:09.514561 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 17:53:09.514801 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:09.515344 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:09.516658 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:09.516871 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:09.516939 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:09.517244 (+ 305us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 17:53:09.517335 (+ 91us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 17:53:09.517386 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0929 17:53:09.517483 (+ 97us) server_negotiation.cc:299] Negotiation successful
0929 17:53:09.517525 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260929 17:53:09.517982 2696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:09.513933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37663 (local address 127.0.98.2:44911)
0929 17:53:09.514250 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:09.514266 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 17:53:09.514366 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:53:09.514571 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:53:09.514574 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:53:09.514581 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 17:53:09.514713 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:09.514719 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:09.515581 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:09.515585 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:09.516555 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:09.516565 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:09.516956 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:09.516960 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:09.517082 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:09.517134 (+ 52us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 17:53:09.517406 (+ 272us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 17:53:09.517412 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 17:53:09.517478 (+ 66us) client_negotiation.cc:240] Negotiation successful
0929 17:53:09.517550 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":105,"threads_started":1}
I20260929 17:53:09.538455 2142 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57472:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 17:53:09.538657 2142 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57472:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 17:53:09.541277 2142 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 72f917d01d69473394db09e8e813365f: Sending DeleteTablet for 3 replicas of tablet f68fd58dffbf4d02bd8b5e8769f9f723
I20260929 17:53:09.542073 2280 tablet_service.cc:1514] Processing DeleteTablet for tablet f68fd58dffbf4d02bd8b5e8769f9f723 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 17:53:09 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59114
I20260929 17:53:09.542186 2560 tablet_service.cc:1514] Processing DeleteTablet for tablet f68fd58dffbf4d02bd8b5e8769f9f723 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 17:53:09 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43038
I20260929 17:53:09.542989 2142 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57472:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260929 17:53:09.543087 2420 tablet_service.cc:1514] Processing DeleteTablet for tablet f68fd58dffbf4d02bd8b5e8769f9f723 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 17:53:09 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54000
W20260929 17:53:09.543411 2142 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 17:53:09.543565 2700 tablet_replica.cc:330] T f68fd58dffbf4d02bd8b5e8769f9f723 P 9e91172c416f417386d7a002c121ca48: stopping tablet replica
I20260929 17:53:09.543898 2700 raft_consensus.cc:2229] T f68fd58dffbf4d02bd8b5e8769f9f723 P 9e91172c416f417386d7a002c121ca48 [term 1 LEADER]: Raft consensus shutting down.
I20260929 17:53:09.544245 2700 raft_consensus.cc:2258] T f68fd58dffbf4d02bd8b5e8769f9f723 P 9e91172c416f417386d7a002c121ca48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 17:53:09.545194 2701 tablet_replica.cc:330] T f68fd58dffbf4d02bd8b5e8769f9f723 P b367e800a8fb4f0aa965712d54ee86f5: stopping tablet replica
I20260929 17:53:09.545555 2701 raft_consensus.cc:2229] T f68fd58dffbf4d02bd8b5e8769f9f723 P b367e800a8fb4f0aa965712d54ee86f5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 17:53:09.545790 2701 raft_consensus.cc:2258] T f68fd58dffbf4d02bd8b5e8769f9f723 P b367e800a8fb4f0aa965712d54ee86f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 17:53:09.547219 2701 ts_tablet_manager.cc:1905] T f68fd58dffbf4d02bd8b5e8769f9f723 P b367e800a8fb4f0aa965712d54ee86f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 17:53:09.549183 2699 tablet_replica.cc:330] T f68fd58dffbf4d02bd8b5e8769f9f723 P 3cddbb0910424ec2b410a25791b1ba46: stopping tablet replica
I20260929 17:53:09.549505 2699 raft_consensus.cc:2229] T f68fd58dffbf4d02bd8b5e8769f9f723 P 3cddbb0910424ec2b410a25791b1ba46 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 17:53:09.549723 2699 raft_consensus.cc:2258] T f68fd58dffbf4d02bd8b5e8769f9f723 P 3cddbb0910424ec2b410a25791b1ba46 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 17:53:09.549912 2280 tablet_service.cc:1467] Processing CreateTablet for tablet 37573d3fa87043e5ba7e97cf6e55fe2c (DEFAULT_TABLE table=test-table [id=9b4725db4c5c454bbb420e47f2dc821f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 17:53:09.550035 2420 tablet_service.cc:1467] Processing CreateTablet for tablet 37573d3fa87043e5ba7e97cf6e55fe2c (DEFAULT_TABLE table=test-table [id=9b4725db4c5c454bbb420e47f2dc821f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 17:53:09.550203 2280 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37573d3fa87043e5ba7e97cf6e55fe2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:53:09.550275 2420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37573d3fa87043e5ba7e97cf6e55fe2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:53:09.551023 2701 ts_tablet_manager.cc:1918] T f68fd58dffbf4d02bd8b5e8769f9f723 P b367e800a8fb4f0aa965712d54ee86f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 17:53:09.551155 2701 log.cc:1196] T f68fd58dffbf4d02bd8b5e8769f9f723 P b367e800a8fb4f0aa965712d54ee86f5: Deleting WAL directory at /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704375975685-392-0/minicluster-data/ts-1/wal/wals/f68fd58dffbf4d02bd8b5e8769f9f723
I20260929 17:53:09.551548 2701 ts_tablet_manager.cc:1939] T f68fd58dffbf4d02bd8b5e8769f9f723 P b367e800a8fb4f0aa965712d54ee86f5: Deleting consensus metadata
I20260929 17:53:09.552392 2649 tablet_bootstrap.cc:492] T 37573d3fa87043e5ba7e97cf6e55fe2c P b367e800a8fb4f0aa965712d54ee86f5: Bootstrap starting.
I20260929 17:53:09.552650 2650 tablet_bootstrap.cc:492] T 37573d3fa87043e5ba7e97cf6e55fe2c P 9e91172c416f417386d7a002c121ca48: Bootstrap starting.
I20260929 17:53:09.552793 2560 tablet_service.cc:1467] Processing CreateTablet for tablet 37573d3fa87043e5ba7e97cf6e55fe2c (DEFAULT_TABLE table=test-table [id=9b4725db4c5c454bbb420e47f2dc821f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 17:53:09.553063 2560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37573d3fa87043e5ba7e97cf6e55fe2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:53:09.553552 2650 tablet_bootstrap.cc:654] T 37573d3fa87043e5ba7e97cf6e55fe2c P 9e91172c416f417386d7a002c121ca48: Neither blocks nor log segments found. Creating new log.
I20260929 17:53:09.553650 2649 tablet_bootstrap.cc:654] T 37573d3fa87043e5ba7e97cf6e55fe2c P b367e800a8fb4f0aa965712d54ee86f5: Neither blocks nor log segments found. Creating new log.
I20260929 17:53:09.555226 2649 tablet_bootstrap.cc:492] T 37573d3fa87043e5ba7e97cf6e55fe2c P b367e800a8fb4f0aa965712d54ee86f5: No bootstrap required, opened a new log
I20260929 17:53:09.555303 2649 ts_tablet_manager.cc:1397] T 37573d3fa87043e5ba7e97cf6e55fe2c P b367e800a8fb4f0aa965712d54ee86f5: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260929 17:53:09.555318 2651 tablet_bootstrap.cc:492] T 37573d3fa87043e5ba7e97cf6e55fe2c P 3cddbb0910424ec2b410a25791b1ba46: Bootstrap starting.
I20260929 17:53:09.555703 2649 raft_consensus.cc:348] T 37573d3fa87043e5ba7e97cf6e55fe2c P b367e800a8fb4f0aa965712d54ee86f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e91172c416f417386d7a002c121ca48" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37663 } } peers { permanent_uuid: "b367e800a8fb4f0aa965712d54ee86f5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46719 } } peers { permanent_uuid: "3cddbb0910424ec2b410a25791b1ba46" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37811 } }
I20260929 17:53:09.555815 2649 raft_consensus.cc:374] T 37573d3fa87043e5ba7e97cf6e55fe2c P b367e800a8fb4f0aa965712d54ee86f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:53:09.555845 2649 raft_consensus.cc:729] T 37573d3fa87043e5ba7e97cf6e55fe2c P b367e800a8fb4f0aa965712d54ee86f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b367e800a8fb4f0aa965712d54ee86f5, State: Initialized, Role: FOLLOWER
I20260929 17:53:09.555953 2649 consensus_queue.cc:260] T 37573d3fa87043e5ba7e97cf6e55fe2c P b367e800a8fb4f0aa965712d54ee86f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e91172c416f417386d7a002c121ca48" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37663 } } peers { permanent_uuid: "b367e800a8fb4f0aa965712d54ee86f5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46719 } } peers { permanent_uuid: "3cddbb0910424ec2b410a25791b1ba46" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37811 } }
I20260929 17:53:09.556167 2651 tablet_bootstrap.cc:654] T 37573d3fa87043e5ba7e97cf6e55fe2c P 3cddbb0910424ec2b410a25791b1ba46: Neither blocks nor log segments found. Creating new log.
I20260929 17:53:09.556178 2649 ts_tablet_manager.cc:1428] T 37573d3fa87043e5ba7e97cf6e55fe2c P b367e800a8fb4f0aa965712d54ee86f5: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260929 17:53:09.556272 2127 catalog_manager.cc:4841] TS b367e800a8fb4f0aa965712d54ee86f5 (127.0.98.2:46719): tablet f68fd58dffbf4d02bd8b5e8769f9f723 (table test-table [id=0592a8beae4743e9bb0123a939e794ff]) successfully deleted
I20260929 17:53:09.557374 2651 tablet_bootstrap.cc:492] T 37573d3fa87043e5ba7e97cf6e55fe2c P 3cddbb0910424ec2b410a25791b1ba46: No bootstrap required, opened a new log
I20260929 17:53:09.557454 2651 ts_tablet_manager.cc:1397] T 37573d3fa87043e5ba7e97cf6e55fe2c P 3cddbb0910424ec2b410a25791b1ba46: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 17:53:09.557857 2651 raft_consensus.cc:348] T 37573d3fa87043e5ba7e97cf6e55fe2c P 3cddbb0910424ec2b410a25791b1ba46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e91172c416f417386d7a002c121ca48" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37663 } } peers { permanent_uuid: "b367e800a8fb4f0aa965712d54ee86f5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46719 } } peers { permanent_uuid: "3cddbb0910424ec2b410a25791b1ba46" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37811 } }
I20260929 17:53:09.557971 2651 raft_consensus.cc:374] T 37573d3fa87043e5ba7e97cf6e55fe2c P 3cddbb0910424ec2b410a25791b1ba46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:53:09.557999 2651 raft_consensus.cc:729] T 37573d3fa87043e5ba7e97cf6e55fe2c P 3cddbb0910424ec2b410a25791b1ba46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3cddbb0910424ec2b410a25791b1ba46, State: Initialized, Role: FOLLOWER
I20260929 17:53:09.558106 2651 consensus_queue.cc:260] T 37573d3fa87043e5ba7e97cf6e55fe2c P 3cddbb0910424ec2b410a25791b1ba46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e91172c416f417386d7a002c121ca48" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37663 } } peers { permanent_uuid: "b367e800a8fb4f0aa965712d54ee86f5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46719 } } peers { permanent_uuid: "3cddbb0910424ec2b410a25791b1ba46" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37811 } }
I20260929 17:53:09.558269 2700 ts_tablet_manager.cc:1905] T f68fd58dffbf4d02bd8b5e8769f9f723 P 9e91172c416f417386d7a002c121ca48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 17:53:09.558341 2651 ts_tablet_manager.cc:1428] T 37573d3fa87043e5ba7e97cf6e55fe2c P 3cddbb0910424ec2b410a25791b1ba46: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 17:53:09.560122 2699 ts_tablet_manager.cc:1905] T f68fd58dffbf4d02bd8b5e8769f9f723 P 3cddbb0910424ec2b410a25791b1ba46: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 17:53:09.561568 2700 ts_tablet_manager.cc:1918] T f68fd58dffbf4d02bd8b5e8769f9f723 P 9e91172c416f417386d7a002c121ca48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 17:53:09.561656 2700 log.cc:1196] T f68fd58dffbf4d02bd8b5e8769f9f723 P 9e91172c416f417386d7a002c121ca48: Deleting WAL directory at /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704375975685-392-0/minicluster-data/ts-0/wal/wals/f68fd58dffbf4d02bd8b5e8769f9f723
I20260929 17:53:09.561673 2650 tablet_bootstrap.cc:492] T 37573d3fa87043e5ba7e97cf6e55fe2c P 9e91172c416f417386d7a002c121ca48: No bootstrap required, opened a new log
I20260929 17:53:09.561758 2650 ts_tablet_manager.cc:1397] T 37573d3fa87043e5ba7e97cf6e55fe2c P 9e91172c416f417386d7a002c121ca48: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.001s
I20260929 17:53:09.561988 2700 ts_tablet_manager.cc:1939] T f68fd58dffbf4d02bd8b5e8769f9f723 P 9e91172c416f417386d7a002c121ca48: Deleting consensus metadata
I20260929 17:53:09.562238 2650 raft_consensus.cc:348] T 37573d3fa87043e5ba7e97cf6e55fe2c P 9e91172c416f417386d7a002c121ca48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e91172c416f417386d7a002c121ca48" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37663 } } peers { permanent_uuid: "b367e800a8fb4f0aa965712d54ee86f5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46719 } } peers { permanent_uuid: "3cddbb0910424ec2b410a25791b1ba46" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37811 } }
I20260929 17:53:09.562366 2650 raft_consensus.cc:374] T 37573d3fa87043e5ba7e97cf6e55fe2c P 9e91172c416f417386d7a002c121ca48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:53:09.562404 2650 raft_consensus.cc:729] T 37573d3fa87043e5ba7e97cf6e55fe2c P 9e91172c416f417386d7a002c121ca48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e91172c416f417386d7a002c121ca48, State: Initialized, Role: FOLLOWER
I20260929 17:53:09.562528 2650 consensus_queue.cc:260] T 37573d3fa87043e5ba7e97cf6e55fe2c P 9e91172c416f417386d7a002c121ca48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e91172c416f417386d7a002c121ca48" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37663 } } peers { permanent_uuid: "b367e800a8fb4f0aa965712d54ee86f5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46719 } } peers { permanent_uuid: "3cddbb0910424ec2b410a25791b1ba46" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37811 } }
I20260929 17:53:09.562927 2699 ts_tablet_manager.cc:1918] T f68fd58dffbf4d02bd8b5e8769f9f723 P 3cddbb0910424ec2b410a25791b1ba46: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 17:53:09.562969 2650 ts_tablet_manager.cc:1428] T 37573d3fa87043e5ba7e97cf6e55fe2c P 9e91172c416f417386d7a002c121ca48: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260929 17:53:09.563012 2699 log.cc:1196] T f68fd58dffbf4d02bd8b5e8769f9f723 P 3cddbb0910424ec2b410a25791b1ba46: Deleting WAL directory at /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704375975685-392-0/minicluster-data/ts-2/wal/wals/f68fd58dffbf4d02bd8b5e8769f9f723
I20260929 17:53:09.563217 2127 catalog_manager.cc:4841] TS 9e91172c416f417386d7a002c121ca48 (127.0.98.1:37663): tablet f68fd58dffbf4d02bd8b5e8769f9f723 (table test-table [id=0592a8beae4743e9bb0123a939e794ff]) successfully deleted
I20260929 17:53:09.563309 2699 ts_tablet_manager.cc:1939] T f68fd58dffbf4d02bd8b5e8769f9f723 P 3cddbb0910424ec2b410a25791b1ba46: Deleting consensus metadata
I20260929 17:53:09.566356 2128 catalog_manager.cc:4841] TS 3cddbb0910424ec2b410a25791b1ba46 (127.0.98.3:37811): tablet f68fd58dffbf4d02bd8b5e8769f9f723 (table test-table [id=0592a8beae4743e9bb0123a939e794ff]) successfully deleted
I20260929 17:53:09.618649 2673 raft_consensus.cc:482] T 37573d3fa87043e5ba7e97cf6e55fe2c P b367e800a8fb4f0aa965712d54ee86f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 17:53:09.618780 2673 raft_consensus.cc:504] T 37573d3fa87043e5ba7e97cf6e55fe2c P b367e800a8fb4f0aa965712d54ee86f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e91172c416f417386d7a002c121ca48" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37663 } } peers { permanent_uuid: "b367e800a8fb4f0aa965712d54ee86f5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46719 } } peers { permanent_uuid: "3cddbb0910424ec2b410a25791b1ba46" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37811 } }
I20260929 17:53:09.619189 2673 leader_election.cc:288] T 37573d3fa87043e5ba7e97cf6e55fe2c P b367e800a8fb4f0aa965712d54ee86f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9e91172c416f417386d7a002c121ca48 (127.0.98.1:37663), 3cddbb0910424ec2b410a25791b1ba46 (127.0.98.3:37811)
I20260929 17:53:09.619552 2300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37573d3fa87043e5ba7e97cf6e55fe2c" candidate_uuid: "b367e800a8fb4f0aa965712d54ee86f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e91172c416f417386d7a002c121ca48" is_pre_election: true
I20260929 17:53:09.619617 2580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37573d3fa87043e5ba7e97cf6e55fe2c" candidate_uuid: "b367e800a8fb4f0aa965712d54ee86f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3cddbb0910424ec2b410a25791b1ba46" is_pre_election: true
I20260929 17:53:09.619701 2300 raft_consensus.cc:2454] T 37573d3fa87043e5ba7e97cf6e55fe2c P 9e91172c416f417386d7a002c121ca48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b367e800a8fb4f0aa965712d54ee86f5 in term 0.
I20260929 17:53:09.619753 2580 raft_consensus.cc:2454] T 37573d3fa87043e5ba7e97cf6e55fe2c P 3cddbb0910424ec2b410a25791b1ba46 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b367e800a8fb4f0aa965712d54ee86f5 in term 0.
I20260929 17:53:09.619994 2371 leader_election.cc:302] T 37573d3fa87043e5ba7e97cf6e55fe2c P b367e800a8fb4f0aa965712d54ee86f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9e91172c416f417386d7a002c121ca48, b367e800a8fb4f0aa965712d54ee86f5; no voters:
I20260929 17:53:09.620213 2673 raft_consensus.cc:2781] T 37573d3fa87043e5ba7e97cf6e55fe2c P b367e800a8fb4f0aa965712d54ee86f5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 17:53:09.620289 2673 raft_consensus.cc:482] T 37573d3fa87043e5ba7e97cf6e55fe2c P b367e800a8fb4f0aa965712d54ee86f5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 17:53:09.620328 2673 raft_consensus.cc:3037] T 37573d3fa87043e5ba7e97cf6e55fe2c P b367e800a8fb4f0aa965712d54ee86f5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:53:09.621340 2673 raft_consensus.cc:504] T 37573d3fa87043e5ba7e97cf6e55fe2c P b367e800a8fb4f0aa965712d54ee86f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e91172c416f417386d7a002c121ca48" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37663 } } peers { permanent_uuid: "b367e800a8fb4f0aa965712d54ee86f5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46719 } } peers { permanent_uuid: "3cddbb0910424ec2b410a25791b1ba46" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37811 } }
I20260929 17:53:09.621644 2673 leader_election.cc:288] T 37573d3fa87043e5ba7e97cf6e55fe2c P b367e800a8fb4f0aa965712d54ee86f5 [CANDIDATE]: Term 1 election: Requested vote from peers 9e91172c416f417386d7a002c121ca48 (127.0.98.1:37663), 3cddbb0910424ec2b410a25791b1ba46 (127.0.98.3:37811)
I20260929 17:53:09.621985 2580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37573d3fa87043e5ba7e97cf6e55fe2c" candidate_uuid: "b367e800a8fb4f0aa965712d54ee86f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3cddbb0910424ec2b410a25791b1ba46"
I20260929 17:53:09.621989 2300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37573d3fa87043e5ba7e97cf6e55fe2c" candidate_uuid: "b367e800a8fb4f0aa965712d54ee86f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e91172c416f417386d7a002c121ca48"
I20260929 17:53:09.622097 2300 raft_consensus.cc:3037] T 37573d3fa87043e5ba7e97cf6e55fe2c P 9e91172c416f417386d7a002c121ca48 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:53:09.622097 2580 raft_consensus.cc:3037] T 37573d3fa87043e5ba7e97cf6e55fe2c P 3cddbb0910424ec2b410a25791b1ba46 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:53:09.622934 2580 raft_consensus.cc:2454] T 37573d3fa87043e5ba7e97cf6e55fe2c P 3cddbb0910424ec2b410a25791b1ba46 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b367e800a8fb4f0aa965712d54ee86f5 in term 1.
I20260929 17:53:09.623152 2300 raft_consensus.cc:2454] T 37573d3fa87043e5ba7e97cf6e55fe2c P 9e91172c416f417386d7a002c121ca48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b367e800a8fb4f0aa965712d54ee86f5 in term 1.
I20260929 17:53:09.623205 2372 leader_election.cc:302] T 37573d3fa87043e5ba7e97cf6e55fe2c P b367e800a8fb4f0aa965712d54ee86f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3cddbb0910424ec2b410a25791b1ba46, b367e800a8fb4f0aa965712d54ee86f5; no voters:
I20260929 17:53:09.623376 2673 raft_consensus.cc:2781] T 37573d3fa87043e5ba7e97cf6e55fe2c P b367e800a8fb4f0aa965712d54ee86f5 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 17:53:09.623462 2673 raft_consensus.cc:686] T 37573d3fa87043e5ba7e97cf6e55fe2c P b367e800a8fb4f0aa965712d54ee86f5 [term 1 LEADER]: Becoming Leader. State: Replica: b367e800a8fb4f0aa965712d54ee86f5, State: Running, Role: LEADER
I20260929 17:53:09.623592 2673 consensus_queue.cc:237] T 37573d3fa87043e5ba7e97cf6e55fe2c P b367e800a8fb4f0aa965712d54ee86f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e91172c416f417386d7a002c121ca48" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37663 } } peers { permanent_uuid: "b367e800a8fb4f0aa965712d54ee86f5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46719 } } peers { permanent_uuid: "3cddbb0910424ec2b410a25791b1ba46" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37811 } }
I20260929 17:53:09.625284 2142 catalog_manager.cc:5495] T 37573d3fa87043e5ba7e97cf6e55fe2c P b367e800a8fb4f0aa965712d54ee86f5 reported cstate change: term changed from 0 to 1, leader changed from <none> to b367e800a8fb4f0aa965712d54ee86f5 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "b367e800a8fb4f0aa965712d54ee86f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e91172c416f417386d7a002c121ca48" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37663 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b367e800a8fb4f0aa965712d54ee86f5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46719 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3cddbb0910424ec2b410a25791b1ba46" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37811 } health_report { overall_health: UNKNOWN } } }
I20260929 17:53:09.647127 2648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:09.644266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54028 (local address 127.0.98.2:46719)
0929 17:53:09.644372 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:09.644376 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:09.644425 (+ 49us) server_negotiation.cc:407] Connection header received
0929 17:53:09.644563 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:09.644567 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:09.644609 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:09.644667 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 17:53:09.644883 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:09.645436 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:09.646217 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:09.646427 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:09.646511 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:09.646738 (+ 227us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 17:53:09.646806 (+ 68us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 17:53:09.646862 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0929 17:53:09.646949 (+ 87us) server_negotiation.cc:299] Negotiation successful
0929 17:53:09.646994 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260929 17:53:09.655489 2648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:09.652859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60531 (local address 127.0.98.2:46719)
0929 17:53:09.652950 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:09.652954 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:09.652965 (+ 11us) server_negotiation.cc:407] Connection header received
0929 17:53:09.653083 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:09.653086 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:09.653124 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:09.653177 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 17:53:09.653402 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:09.653874 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:09.654613 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:09.654812 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:09.654891 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:09.655079 (+ 188us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 17:53:09.655138 (+ 59us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 17:53:09.655192 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0929 17:53:09.655270 (+ 78us) server_negotiation.cc:299] Negotiation successful
0929 17:53:09.655331 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":24,"server-negotiator.queue_time_us":39}
I20260929 17:53:09.655489 2696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:09.652788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46719 (local address 127.0.98.2:60531)
0929 17:53:09.652895 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:09.652910 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 17:53:09.652984 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:53:09.653189 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:53:09.653192 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:53:09.653198 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 17:53:09.653322 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:09.653327 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:09.653961 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:09.653964 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:09.654521 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:09.654528 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:09.654898 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:09.654901 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:09.654973 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:09.654993 (+ 20us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 17:53:09.655202 (+ 209us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 17:53:09.655207 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 17:53:09.655263 (+ 56us) client_negotiation.cc:240] Negotiation successful
0929 17:53:09.655356 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25,"mutex_wait_us":20}
I20260929 17:53:09.663831 2580 raft_consensus.cc:1261] T 37573d3fa87043e5ba7e97cf6e55fe2c P 3cddbb0910424ec2b410a25791b1ba46 [term 1 FOLLOWER]: Refusing update from remote peer b367e800a8fb4f0aa965712d54ee86f5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 17:53:09.664412 2691 consensus_queue.cc:1035] T 37573d3fa87043e5ba7e97cf6e55fe2c P b367e800a8fb4f0aa965712d54ee86f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3cddbb0910424ec2b410a25791b1ba46" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37811 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 17:53:09.664536 2300 raft_consensus.cc:1261] T 37573d3fa87043e5ba7e97cf6e55fe2c P 9e91172c416f417386d7a002c121ca48 [term 1 FOLLOWER]: Refusing update from remote peer b367e800a8fb4f0aa965712d54ee86f5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 17:53:09.665069 2691 consensus_queue.cc:1035] T 37573d3fa87043e5ba7e97cf6e55fe2c P b367e800a8fb4f0aa965712d54ee86f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e91172c416f417386d7a002c121ca48" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37663 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260929 17:53:09.681335 2678 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260929 17:53:11.235067 2143 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57472:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 17:53:11.235224 2143 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57472:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 17:53:11.237581 2143 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 72f917d01d69473394db09e8e813365f: Sending DeleteTablet for 3 replicas of tablet 37573d3fa87043e5ba7e97cf6e55fe2c
I20260929 17:53:11.238106 2280 tablet_service.cc:1514] Processing DeleteTablet for tablet 37573d3fa87043e5ba7e97cf6e55fe2c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 17:53:11 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59114
I20260929 17:53:11.238161 2560 tablet_service.cc:1514] Processing DeleteTablet for tablet 37573d3fa87043e5ba7e97cf6e55fe2c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 17:53:11 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43038
I20260929 17:53:11.238153 2420 tablet_service.cc:1514] Processing DeleteTablet for tablet 37573d3fa87043e5ba7e97cf6e55fe2c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 17:53:11 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54000
I20260929 17:53:11.238456 2728 tablet_replica.cc:330] T 37573d3fa87043e5ba7e97cf6e55fe2c P 9e91172c416f417386d7a002c121ca48: stopping tablet replica
I20260929 17:53:11.238488 2730 tablet_replica.cc:330] T 37573d3fa87043e5ba7e97cf6e55fe2c P b367e800a8fb4f0aa965712d54ee86f5: stopping tablet replica
I20260929 17:53:11.238584 2728 raft_consensus.cc:2229] T 37573d3fa87043e5ba7e97cf6e55fe2c P 9e91172c416f417386d7a002c121ca48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 17:53:11.238596 2730 raft_consensus.cc:2229] T 37573d3fa87043e5ba7e97cf6e55fe2c P b367e800a8fb4f0aa965712d54ee86f5 [term 1 LEADER]: Raft consensus shutting down.
I20260929 17:53:11.238708 2728 raft_consensus.cc:2258] T 37573d3fa87043e5ba7e97cf6e55fe2c P 9e91172c416f417386d7a002c121ca48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 17:53:11.239217 2730 raft_consensus.cc:2258] T 37573d3fa87043e5ba7e97cf6e55fe2c P b367e800a8fb4f0aa965712d54ee86f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 17:53:11.239750 2728 ts_tablet_manager.cc:1905] T 37573d3fa87043e5ba7e97cf6e55fe2c P 9e91172c416f417386d7a002c121ca48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 17:53:11.241575 2730 ts_tablet_manager.cc:1905] T 37573d3fa87043e5ba7e97cf6e55fe2c P b367e800a8fb4f0aa965712d54ee86f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 17:53:11.241616 2729 tablet_replica.cc:330] T 37573d3fa87043e5ba7e97cf6e55fe2c P 3cddbb0910424ec2b410a25791b1ba46: stopping tablet replica
I20260929 17:53:11.241741 2729 raft_consensus.cc:2229] T 37573d3fa87043e5ba7e97cf6e55fe2c P 3cddbb0910424ec2b410a25791b1ba46 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 17:53:11.241868 2729 raft_consensus.cc:2258] T 37573d3fa87043e5ba7e97cf6e55fe2c P 3cddbb0910424ec2b410a25791b1ba46 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 17:53:11.242437 2729 ts_tablet_manager.cc:1905] T 37573d3fa87043e5ba7e97cf6e55fe2c P 3cddbb0910424ec2b410a25791b1ba46: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 17:53:11.243721 2730 ts_tablet_manager.cc:1918] T 37573d3fa87043e5ba7e97cf6e55fe2c P b367e800a8fb4f0aa965712d54ee86f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 17:53:11.243789 2730 log.cc:1196] T 37573d3fa87043e5ba7e97cf6e55fe2c P b367e800a8fb4f0aa965712d54ee86f5: Deleting WAL directory at /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704375975685-392-0/minicluster-data/ts-1/wal/wals/37573d3fa87043e5ba7e97cf6e55fe2c
I20260929 17:53:11.244029 2730 ts_tablet_manager.cc:1939] T 37573d3fa87043e5ba7e97cf6e55fe2c P b367e800a8fb4f0aa965712d54ee86f5: Deleting consensus metadata
I20260929 17:53:11.244648 2127 catalog_manager.cc:4841] TS b367e800a8fb4f0aa965712d54ee86f5 (127.0.98.2:46719): tablet 37573d3fa87043e5ba7e97cf6e55fe2c (table test-table [id=9b4725db4c5c454bbb420e47f2dc821f]) successfully deleted
I20260929 17:53:11.244952 2729 ts_tablet_manager.cc:1918] T 37573d3fa87043e5ba7e97cf6e55fe2c P 3cddbb0910424ec2b410a25791b1ba46: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 17:53:11.245051 2729 log.cc:1196] T 37573d3fa87043e5ba7e97cf6e55fe2c P 3cddbb0910424ec2b410a25791b1ba46: Deleting WAL directory at /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704375975685-392-0/minicluster-data/ts-2/wal/wals/37573d3fa87043e5ba7e97cf6e55fe2c
I20260929 17:53:11.245159 2728 ts_tablet_manager.cc:1918] T 37573d3fa87043e5ba7e97cf6e55fe2c P 9e91172c416f417386d7a002c121ca48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 17:53:11.245232 2728 log.cc:1196] T 37573d3fa87043e5ba7e97cf6e55fe2c P 9e91172c416f417386d7a002c121ca48: Deleting WAL directory at /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704375975685-392-0/minicluster-data/ts-0/wal/wals/37573d3fa87043e5ba7e97cf6e55fe2c
I20260929 17:53:11.245280 2729 ts_tablet_manager.cc:1939] T 37573d3fa87043e5ba7e97cf6e55fe2c P 3cddbb0910424ec2b410a25791b1ba46: Deleting consensus metadata
I20260929 17:53:11.245493 2728 ts_tablet_manager.cc:1939] T 37573d3fa87043e5ba7e97cf6e55fe2c P 9e91172c416f417386d7a002c121ca48: Deleting consensus metadata
I20260929 17:53:11.245931 2128 catalog_manager.cc:4841] TS 3cddbb0910424ec2b410a25791b1ba46 (127.0.98.3:37811): tablet 37573d3fa87043e5ba7e97cf6e55fe2c (table test-table [id=9b4725db4c5c454bbb420e47f2dc821f]) successfully deleted
I20260929 17:53:11.246075 2127 catalog_manager.cc:4841] TS 9e91172c416f417386d7a002c121ca48 (127.0.98.1:37663): tablet 37573d3fa87043e5ba7e97cf6e55fe2c (table test-table [id=9b4725db4c5c454bbb420e47f2dc821f]) successfully deleted
Sep 29 17:53:11 dist-test-slave-53nh krb5kdc[2098](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704391, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 17:53:11 dist-test-slave-53nh krb5kdc[2098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704391, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 17:53:11.272841 2738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:11.258035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57484 (local address 127.0.98.62:42197)
0929 17:53:11.258327 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:11.258334 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:11.258350 (+ 16us) server_negotiation.cc:407] Connection header received
0929 17:53:11.258476 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:11.258480 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:11.258532 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:11.258616 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:53:11.259286 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:11.259847 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:11.260605 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:11.260844 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:11.260936 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:11.269773 (+ 8837us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:53:11.269777 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:53:11.269779 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:53:11.269812 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:53:11.271169 (+ 1357us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:53:11.271615 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:53:11.271619 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:53:11.271621 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:53:11.271709 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:53:11.271957 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:53:11.271960 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:53:11.271962 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:53:11.272153 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:53:11.272276 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0929 17:53:11.272537 (+ 261us) server_negotiation.cc:299] Negotiation successful
0929 17:53:11.272664 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":218,"thread_start_us":104,"threads_started":1}
I20260929 17:53:11.275370 2143 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57484:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 17:53:11.275790 2143 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 17:53:11.280105 2280 tablet_service.cc:1467] Processing CreateTablet for tablet 60a734b549874b46b88a73d4a059b2f8 (DEFAULT_TABLE table=test-table [id=d9ffa6db44a44ca7be76454e80a69949]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 17:53:11.280082 2420 tablet_service.cc:1467] Processing CreateTablet for tablet 60a734b549874b46b88a73d4a059b2f8 (DEFAULT_TABLE table=test-table [id=d9ffa6db44a44ca7be76454e80a69949]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 17:53:11.280380 2280 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60a734b549874b46b88a73d4a059b2f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:53:11.280380 2420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60a734b549874b46b88a73d4a059b2f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:53:11.281446 2560 tablet_service.cc:1467] Processing CreateTablet for tablet 60a734b549874b46b88a73d4a059b2f8 (DEFAULT_TABLE table=test-table [id=d9ffa6db44a44ca7be76454e80a69949]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 17:53:11.281713 2560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60a734b549874b46b88a73d4a059b2f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:53:11.282953 2740 tablet_bootstrap.cc:492] T 60a734b549874b46b88a73d4a059b2f8 P 9e91172c416f417386d7a002c121ca48: Bootstrap starting.
I20260929 17:53:11.283077 2739 tablet_bootstrap.cc:492] T 60a734b549874b46b88a73d4a059b2f8 P b367e800a8fb4f0aa965712d54ee86f5: Bootstrap starting.
I20260929 17:53:11.284000 2741 tablet_bootstrap.cc:492] T 60a734b549874b46b88a73d4a059b2f8 P 3cddbb0910424ec2b410a25791b1ba46: Bootstrap starting.
I20260929 17:53:11.284124 2740 tablet_bootstrap.cc:654] T 60a734b549874b46b88a73d4a059b2f8 P 9e91172c416f417386d7a002c121ca48: Neither blocks nor log segments found. Creating new log.
I20260929 17:53:11.284189 2739 tablet_bootstrap.cc:654] T 60a734b549874b46b88a73d4a059b2f8 P b367e800a8fb4f0aa965712d54ee86f5: Neither blocks nor log segments found. Creating new log.
I20260929 17:53:11.284871 2741 tablet_bootstrap.cc:654] T 60a734b549874b46b88a73d4a059b2f8 P 3cddbb0910424ec2b410a25791b1ba46: Neither blocks nor log segments found. Creating new log.
I20260929 17:53:11.285104 2740 tablet_bootstrap.cc:492] T 60a734b549874b46b88a73d4a059b2f8 P 9e91172c416f417386d7a002c121ca48: No bootstrap required, opened a new log
I20260929 17:53:11.285180 2740 ts_tablet_manager.cc:1397] T 60a734b549874b46b88a73d4a059b2f8 P 9e91172c416f417386d7a002c121ca48: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260929 17:53:11.285539 2740 raft_consensus.cc:348] T 60a734b549874b46b88a73d4a059b2f8 P 9e91172c416f417386d7a002c121ca48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e91172c416f417386d7a002c121ca48" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37663 } } peers { permanent_uuid: "b367e800a8fb4f0aa965712d54ee86f5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46719 } } peers { permanent_uuid: "3cddbb0910424ec2b410a25791b1ba46" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37811 } }
I20260929 17:53:11.285655 2740 raft_consensus.cc:374] T 60a734b549874b46b88a73d4a059b2f8 P 9e91172c416f417386d7a002c121ca48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:53:11.285696 2740 raft_consensus.cc:729] T 60a734b549874b46b88a73d4a059b2f8 P 9e91172c416f417386d7a002c121ca48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e91172c416f417386d7a002c121ca48, State: Initialized, Role: FOLLOWER
I20260929 17:53:11.285821 2740 consensus_queue.cc:260] T 60a734b549874b46b88a73d4a059b2f8 P 9e91172c416f417386d7a002c121ca48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e91172c416f417386d7a002c121ca48" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37663 } } peers { permanent_uuid: "b367e800a8fb4f0aa965712d54ee86f5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46719 } } peers { permanent_uuid: "3cddbb0910424ec2b410a25791b1ba46" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37811 } }
I20260929 17:53:11.285980 2741 tablet_bootstrap.cc:492] T 60a734b549874b46b88a73d4a059b2f8 P 3cddbb0910424ec2b410a25791b1ba46: No bootstrap required, opened a new log
I20260929 17:53:11.286058 2741 ts_tablet_manager.cc:1397] T 60a734b549874b46b88a73d4a059b2f8 P 3cddbb0910424ec2b410a25791b1ba46: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260929 17:53:11.286168 2740 ts_tablet_manager.cc:1428] T 60a734b549874b46b88a73d4a059b2f8 P 9e91172c416f417386d7a002c121ca48: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 17:53:11.286773 2739 tablet_bootstrap.cc:492] T 60a734b549874b46b88a73d4a059b2f8 P b367e800a8fb4f0aa965712d54ee86f5: No bootstrap required, opened a new log
I20260929 17:53:11.286855 2739 ts_tablet_manager.cc:1397] T 60a734b549874b46b88a73d4a059b2f8 P b367e800a8fb4f0aa965712d54ee86f5: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260929 17:53:11.286912 2741 raft_consensus.cc:348] T 60a734b549874b46b88a73d4a059b2f8 P 3cddbb0910424ec2b410a25791b1ba46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e91172c416f417386d7a002c121ca48" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37663 } } peers { permanent_uuid: "b367e800a8fb4f0aa965712d54ee86f5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46719 } } peers { permanent_uuid: "3cddbb0910424ec2b410a25791b1ba46" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37811 } }
I20260929 17:53:11.287010 2741 raft_consensus.cc:374] T 60a734b549874b46b88a73d4a059b2f8 P 3cddbb0910424ec2b410a25791b1ba46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:53:11.287040 2741 raft_consensus.cc:729] T 60a734b549874b46b88a73d4a059b2f8 P 3cddbb0910424ec2b410a25791b1ba46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3cddbb0910424ec2b410a25791b1ba46, State: Initialized, Role: FOLLOWER
I20260929 17:53:11.287163 2739 raft_consensus.cc:348] T 60a734b549874b46b88a73d4a059b2f8 P b367e800a8fb4f0aa965712d54ee86f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e91172c416f417386d7a002c121ca48" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37663 } } peers { permanent_uuid: "b367e800a8fb4f0aa965712d54ee86f5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46719 } } peers { permanent_uuid: "3cddbb0910424ec2b410a25791b1ba46" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37811 } }
I20260929 17:53:11.287164 2741 consensus_queue.cc:260] T 60a734b549874b46b88a73d4a059b2f8 P 3cddbb0910424ec2b410a25791b1ba46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e91172c416f417386d7a002c121ca48" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37663 } } peers { permanent_uuid: "b367e800a8fb4f0aa965712d54ee86f5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46719 } } peers { permanent_uuid: "3cddbb0910424ec2b410a25791b1ba46" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37811 } }
I20260929 17:53:11.287258 2739 raft_consensus.cc:374] T 60a734b549874b46b88a73d4a059b2f8 P b367e800a8fb4f0aa965712d54ee86f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:53:11.287278 2739 raft_consensus.cc:729] T 60a734b549874b46b88a73d4a059b2f8 P b367e800a8fb4f0aa965712d54ee86f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b367e800a8fb4f0aa965712d54ee86f5, State: Initialized, Role: FOLLOWER
I20260929 17:53:11.287374 2739 consensus_queue.cc:260] T 60a734b549874b46b88a73d4a059b2f8 P b367e800a8fb4f0aa965712d54ee86f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e91172c416f417386d7a002c121ca48" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37663 } } peers { permanent_uuid: "b367e800a8fb4f0aa965712d54ee86f5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46719 } } peers { permanent_uuid: "3cddbb0910424ec2b410a25791b1ba46" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37811 } }
I20260929 17:53:11.287611 2739 ts_tablet_manager.cc:1428] T 60a734b549874b46b88a73d4a059b2f8 P b367e800a8fb4f0aa965712d54ee86f5: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260929 17:53:11.287892 2741 ts_tablet_manager.cc:1428] T 60a734b549874b46b88a73d4a059b2f8 P 3cddbb0910424ec2b410a25791b1ba46: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260929 17:53:11.353878 2673 raft_consensus.cc:482] T 60a734b549874b46b88a73d4a059b2f8 P b367e800a8fb4f0aa965712d54ee86f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 17:53:11.354017 2673 raft_consensus.cc:504] T 60a734b549874b46b88a73d4a059b2f8 P b367e800a8fb4f0aa965712d54ee86f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e91172c416f417386d7a002c121ca48" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37663 } } peers { permanent_uuid: "b367e800a8fb4f0aa965712d54ee86f5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46719 } } peers { permanent_uuid: "3cddbb0910424ec2b410a25791b1ba46" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37811 } }
I20260929 17:53:11.354408 2673 leader_election.cc:288] T 60a734b549874b46b88a73d4a059b2f8 P b367e800a8fb4f0aa965712d54ee86f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9e91172c416f417386d7a002c121ca48 (127.0.98.1:37663), 3cddbb0910424ec2b410a25791b1ba46 (127.0.98.3:37811)
I20260929 17:53:11.354821 2580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60a734b549874b46b88a73d4a059b2f8" candidate_uuid: "b367e800a8fb4f0aa965712d54ee86f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3cddbb0910424ec2b410a25791b1ba46" is_pre_election: true
I20260929 17:53:11.354852 2299 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60a734b549874b46b88a73d4a059b2f8" candidate_uuid: "b367e800a8fb4f0aa965712d54ee86f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e91172c416f417386d7a002c121ca48" is_pre_election: true
I20260929 17:53:11.354950 2580 raft_consensus.cc:2454] T 60a734b549874b46b88a73d4a059b2f8 P 3cddbb0910424ec2b410a25791b1ba46 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b367e800a8fb4f0aa965712d54ee86f5 in term 0.
I20260929 17:53:11.355022 2299 raft_consensus.cc:2454] T 60a734b549874b46b88a73d4a059b2f8 P 9e91172c416f417386d7a002c121ca48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b367e800a8fb4f0aa965712d54ee86f5 in term 0.
I20260929 17:53:11.355285 2372 leader_election.cc:302] T 60a734b549874b46b88a73d4a059b2f8 P b367e800a8fb4f0aa965712d54ee86f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3cddbb0910424ec2b410a25791b1ba46, b367e800a8fb4f0aa965712d54ee86f5; no voters:
I20260929 17:53:11.355432 2673 raft_consensus.cc:2781] T 60a734b549874b46b88a73d4a059b2f8 P b367e800a8fb4f0aa965712d54ee86f5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 17:53:11.355494 2673 raft_consensus.cc:482] T 60a734b549874b46b88a73d4a059b2f8 P b367e800a8fb4f0aa965712d54ee86f5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 17:53:11.355520 2673 raft_consensus.cc:3037] T 60a734b549874b46b88a73d4a059b2f8 P b367e800a8fb4f0aa965712d54ee86f5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:53:11.356484 2673 raft_consensus.cc:504] T 60a734b549874b46b88a73d4a059b2f8 P b367e800a8fb4f0aa965712d54ee86f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e91172c416f417386d7a002c121ca48" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37663 } } peers { permanent_uuid: "b367e800a8fb4f0aa965712d54ee86f5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46719 } } peers { permanent_uuid: "3cddbb0910424ec2b410a25791b1ba46" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37811 } }
I20260929 17:53:11.356799 2673 leader_election.cc:288] T 60a734b549874b46b88a73d4a059b2f8 P b367e800a8fb4f0aa965712d54ee86f5 [CANDIDATE]: Term 1 election: Requested vote from peers 9e91172c416f417386d7a002c121ca48 (127.0.98.1:37663), 3cddbb0910424ec2b410a25791b1ba46 (127.0.98.3:37811)
I20260929 17:53:11.357139 2299 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60a734b549874b46b88a73d4a059b2f8" candidate_uuid: "b367e800a8fb4f0aa965712d54ee86f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e91172c416f417386d7a002c121ca48"
I20260929 17:53:11.357182 2580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60a734b549874b46b88a73d4a059b2f8" candidate_uuid: "b367e800a8fb4f0aa965712d54ee86f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3cddbb0910424ec2b410a25791b1ba46"
I20260929 17:53:11.357286 2580 raft_consensus.cc:3037] T 60a734b549874b46b88a73d4a059b2f8 P 3cddbb0910424ec2b410a25791b1ba46 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:53:11.357286 2299 raft_consensus.cc:3037] T 60a734b549874b46b88a73d4a059b2f8 P 9e91172c416f417386d7a002c121ca48 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:53:11.358423 2299 raft_consensus.cc:2454] T 60a734b549874b46b88a73d4a059b2f8 P 9e91172c416f417386d7a002c121ca48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b367e800a8fb4f0aa965712d54ee86f5 in term 1.
I20260929 17:53:11.358423 2580 raft_consensus.cc:2454] T 60a734b549874b46b88a73d4a059b2f8 P 3cddbb0910424ec2b410a25791b1ba46 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b367e800a8fb4f0aa965712d54ee86f5 in term 1.
I20260929 17:53:11.358695 2372 leader_election.cc:302] T 60a734b549874b46b88a73d4a059b2f8 P b367e800a8fb4f0aa965712d54ee86f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3cddbb0910424ec2b410a25791b1ba46, b367e800a8fb4f0aa965712d54ee86f5; no voters:
I20260929 17:53:11.358871 2673 raft_consensus.cc:2781] T 60a734b549874b46b88a73d4a059b2f8 P b367e800a8fb4f0aa965712d54ee86f5 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 17:53:11.358971 2673 raft_consensus.cc:686] T 60a734b549874b46b88a73d4a059b2f8 P b367e800a8fb4f0aa965712d54ee86f5 [term 1 LEADER]: Becoming Leader. State: Replica: b367e800a8fb4f0aa965712d54ee86f5, State: Running, Role: LEADER
I20260929 17:53:11.359175 2673 consensus_queue.cc:237] T 60a734b549874b46b88a73d4a059b2f8 P b367e800a8fb4f0aa965712d54ee86f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e91172c416f417386d7a002c121ca48" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37663 } } peers { permanent_uuid: "b367e800a8fb4f0aa965712d54ee86f5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46719 } } peers { permanent_uuid: "3cddbb0910424ec2b410a25791b1ba46" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37811 } }
I20260929 17:53:11.360874 2143 catalog_manager.cc:5495] T 60a734b549874b46b88a73d4a059b2f8 P b367e800a8fb4f0aa965712d54ee86f5 reported cstate change: term changed from 0 to 1, leader changed from <none> to b367e800a8fb4f0aa965712d54ee86f5 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "b367e800a8fb4f0aa965712d54ee86f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e91172c416f417386d7a002c121ca48" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37663 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b367e800a8fb4f0aa965712d54ee86f5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46719 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3cddbb0910424ec2b410a25791b1ba46" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37811 } health_report { overall_health: UNKNOWN } } }
I20260929 17:53:11.388770 2749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:11.385740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54042 (local address 127.0.98.2:46719)
0929 17:53:11.385995 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:11.386003 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:11.386015 (+ 12us) server_negotiation.cc:407] Connection header received
0929 17:53:11.386053 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:11.386056 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:11.386097 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:11.386162 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 17:53:11.386411 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:11.386944 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:11.387757 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:11.388008 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:11.388090 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:11.388349 (+ 259us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 17:53:11.388420 (+ 71us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 17:53:11.388475 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0929 17:53:11.388571 (+ 96us) server_negotiation.cc:299] Negotiation successful
0929 17:53:11.388620 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":96,"threads_started":1}
I20260929 17:53:11.394193 2300 raft_consensus.cc:1261] T 60a734b549874b46b88a73d4a059b2f8 P 9e91172c416f417386d7a002c121ca48 [term 1 FOLLOWER]: Refusing update from remote peer b367e800a8fb4f0aa965712d54ee86f5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 17:53:11.394240 2580 raft_consensus.cc:1261] T 60a734b549874b46b88a73d4a059b2f8 P 3cddbb0910424ec2b410a25791b1ba46 [term 1 FOLLOWER]: Refusing update from remote peer b367e800a8fb4f0aa965712d54ee86f5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 17:53:11.394538 2711 consensus_queue.cc:1035] T 60a734b549874b46b88a73d4a059b2f8 P b367e800a8fb4f0aa965712d54ee86f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3cddbb0910424ec2b410a25791b1ba46" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37811 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 17:53:11.394701 2673 consensus_queue.cc:1035] T 60a734b549874b46b88a73d4a059b2f8 P b367e800a8fb4f0aa965712d54ee86f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e91172c416f417386d7a002c121ca48" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37663 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 17:53:11.422478 2143 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57484:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 17:53:11.422612 2143 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57484:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 17:53:11.424368 2143 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 72f917d01d69473394db09e8e813365f: Sending DeleteTablet for 3 replicas of tablet 60a734b549874b46b88a73d4a059b2f8
I20260929 17:53:11.424822 2280 tablet_service.cc:1514] Processing DeleteTablet for tablet 60a734b549874b46b88a73d4a059b2f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 17:53:11 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59114
I20260929 17:53:11.424942 2420 tablet_service.cc:1514] Processing DeleteTablet for tablet 60a734b549874b46b88a73d4a059b2f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 17:53:11 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54000
I20260929 17:53:11.425073 2728 tablet_replica.cc:330] T 60a734b549874b46b88a73d4a059b2f8 P 9e91172c416f417386d7a002c121ca48: stopping tablet replica
I20260929 17:53:11.425076 2560 tablet_service.cc:1514] Processing DeleteTablet for tablet 60a734b549874b46b88a73d4a059b2f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 17:53:11 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43038
I20260929 17:53:11.425215 2728 raft_consensus.cc:2229] T 60a734b549874b46b88a73d4a059b2f8 P 9e91172c416f417386d7a002c121ca48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 17:53:11.425269 2730 tablet_replica.cc:330] T 60a734b549874b46b88a73d4a059b2f8 P b367e800a8fb4f0aa965712d54ee86f5: stopping tablet replica
I20260929 17:53:11.425239 2729 tablet_replica.cc:330] T 60a734b549874b46b88a73d4a059b2f8 P 3cddbb0910424ec2b410a25791b1ba46: stopping tablet replica
I20260929 17:53:11.425329 2728 raft_consensus.cc:2258] T 60a734b549874b46b88a73d4a059b2f8 P 9e91172c416f417386d7a002c121ca48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 17:53:11.425366 2730 raft_consensus.cc:2229] T 60a734b549874b46b88a73d4a059b2f8 P b367e800a8fb4f0aa965712d54ee86f5 [term 1 LEADER]: Raft consensus shutting down.
I20260929 17:53:11.425388 2729 raft_consensus.cc:2229] T 60a734b549874b46b88a73d4a059b2f8 P 3cddbb0910424ec2b410a25791b1ba46 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 17:53:11.425498 2729 raft_consensus.cc:2258] T 60a734b549874b46b88a73d4a059b2f8 P 3cddbb0910424ec2b410a25791b1ba46 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 17:53:11.425510 2730 raft_consensus.cc:2258] T 60a734b549874b46b88a73d4a059b2f8 P b367e800a8fb4f0aa965712d54ee86f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 17:53:11.425591 2143 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57484:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 17:53:11.425979 2143 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 17:53:11.426227 2728 ts_tablet_manager.cc:1905] T 60a734b549874b46b88a73d4a059b2f8 P 9e91172c416f417386d7a002c121ca48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 17:53:11.426316 2730 ts_tablet_manager.cc:1905] T 60a734b549874b46b88a73d4a059b2f8 P b367e800a8fb4f0aa965712d54ee86f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 17:53:11.428319 2728 ts_tablet_manager.cc:1918] T 60a734b549874b46b88a73d4a059b2f8 P 9e91172c416f417386d7a002c121ca48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 17:53:11.428323 2730 ts_tablet_manager.cc:1918] T 60a734b549874b46b88a73d4a059b2f8 P b367e800a8fb4f0aa965712d54ee86f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 17:53:11.428380 2730 log.cc:1196] T 60a734b549874b46b88a73d4a059b2f8 P b367e800a8fb4f0aa965712d54ee86f5: Deleting WAL directory at /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704375975685-392-0/minicluster-data/ts-1/wal/wals/60a734b549874b46b88a73d4a059b2f8
I20260929 17:53:11.428386 2728 log.cc:1196] T 60a734b549874b46b88a73d4a059b2f8 P 9e91172c416f417386d7a002c121ca48: Deleting WAL directory at /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704375975685-392-0/minicluster-data/ts-0/wal/wals/60a734b549874b46b88a73d4a059b2f8
I20260929 17:53:11.428574 2730 ts_tablet_manager.cc:1939] T 60a734b549874b46b88a73d4a059b2f8 P b367e800a8fb4f0aa965712d54ee86f5: Deleting consensus metadata
I20260929 17:53:11.428588 2728 ts_tablet_manager.cc:1939] T 60a734b549874b46b88a73d4a059b2f8 P 9e91172c416f417386d7a002c121ca48: Deleting consensus metadata
I20260929 17:53:11.429358 2127 catalog_manager.cc:4841] TS b367e800a8fb4f0aa965712d54ee86f5 (127.0.98.2:46719): tablet 60a734b549874b46b88a73d4a059b2f8 (table test-table [id=d9ffa6db44a44ca7be76454e80a69949]) successfully deleted
I20260929 17:53:11.429443 2729 ts_tablet_manager.cc:1905] T 60a734b549874b46b88a73d4a059b2f8 P 3cddbb0910424ec2b410a25791b1ba46: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 17:53:11.430403 2127 catalog_manager.cc:4841] TS 9e91172c416f417386d7a002c121ca48 (127.0.98.1:37663): tablet 60a734b549874b46b88a73d4a059b2f8 (table test-table [id=d9ffa6db44a44ca7be76454e80a69949]) successfully deleted
I20260929 17:53:11.431702 2729 ts_tablet_manager.cc:1918] T 60a734b549874b46b88a73d4a059b2f8 P 3cddbb0910424ec2b410a25791b1ba46: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 17:53:11.431789 2729 log.cc:1196] T 60a734b549874b46b88a73d4a059b2f8 P 3cddbb0910424ec2b410a25791b1ba46: Deleting WAL directory at /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704375975685-392-0/minicluster-data/ts-2/wal/wals/60a734b549874b46b88a73d4a059b2f8
I20260929 17:53:11.431898 2280 tablet_service.cc:1467] Processing CreateTablet for tablet c477d231a7ce4d61b3b472e027daaf9c (DEFAULT_TABLE table=test-table [id=82b1a4883674434eb79d058736614d79]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 17:53:11.431928 2560 tablet_service.cc:1467] Processing CreateTablet for tablet c477d231a7ce4d61b3b472e027daaf9c (DEFAULT_TABLE table=test-table [id=82b1a4883674434eb79d058736614d79]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 17:53:11.432152 2560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c477d231a7ce4d61b3b472e027daaf9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:53:11.432163 2280 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c477d231a7ce4d61b3b472e027daaf9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:53:11.432156 2420 tablet_service.cc:1467] Processing CreateTablet for tablet c477d231a7ce4d61b3b472e027daaf9c (DEFAULT_TABLE table=test-table [id=82b1a4883674434eb79d058736614d79]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 17:53:11.432498 2420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c477d231a7ce4d61b3b472e027daaf9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:53:11.434218 2741 tablet_bootstrap.cc:492] T c477d231a7ce4d61b3b472e027daaf9c P 3cddbb0910424ec2b410a25791b1ba46: Bootstrap starting.
I20260929 17:53:11.434545 2729 ts_tablet_manager.cc:1939] T 60a734b549874b46b88a73d4a059b2f8 P 3cddbb0910424ec2b410a25791b1ba46: Deleting consensus metadata
I20260929 17:53:11.435135 2741 tablet_bootstrap.cc:654] T c477d231a7ce4d61b3b472e027daaf9c P 3cddbb0910424ec2b410a25791b1ba46: Neither blocks nor log segments found. Creating new log.
I20260929 17:53:11.434230 2740 tablet_bootstrap.cc:492] T c477d231a7ce4d61b3b472e027daaf9c P 9e91172c416f417386d7a002c121ca48: Bootstrap starting.
I20260929 17:53:11.436050 2128 catalog_manager.cc:4841] TS 3cddbb0910424ec2b410a25791b1ba46 (127.0.98.3:37811): tablet 60a734b549874b46b88a73d4a059b2f8 (table test-table [id=d9ffa6db44a44ca7be76454e80a69949]) successfully deleted
I20260929 17:53:11.436139 2740 tablet_bootstrap.cc:654] T c477d231a7ce4d61b3b472e027daaf9c P 9e91172c416f417386d7a002c121ca48: Neither blocks nor log segments found. Creating new log.
I20260929 17:53:11.436349 2741 tablet_bootstrap.cc:492] T c477d231a7ce4d61b3b472e027daaf9c P 3cddbb0910424ec2b410a25791b1ba46: No bootstrap required, opened a new log
I20260929 17:53:11.436417 2741 ts_tablet_manager.cc:1397] T c477d231a7ce4d61b3b472e027daaf9c P 3cddbb0910424ec2b410a25791b1ba46: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260929 17:53:11.436836 2741 raft_consensus.cc:348] T c477d231a7ce4d61b3b472e027daaf9c P 3cddbb0910424ec2b410a25791b1ba46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e91172c416f417386d7a002c121ca48" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37663 } } peers { permanent_uuid: "b367e800a8fb4f0aa965712d54ee86f5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46719 } } peers { permanent_uuid: "3cddbb0910424ec2b410a25791b1ba46" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37811 } }
I20260929 17:53:11.436956 2741 raft_consensus.cc:374] T c477d231a7ce4d61b3b472e027daaf9c P 3cddbb0910424ec2b410a25791b1ba46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:53:11.436990 2741 raft_consensus.cc:729] T c477d231a7ce4d61b3b472e027daaf9c P 3cddbb0910424ec2b410a25791b1ba46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3cddbb0910424ec2b410a25791b1ba46, State: Initialized, Role: FOLLOWER
I20260929 17:53:11.437101 2740 tablet_bootstrap.cc:492] T c477d231a7ce4d61b3b472e027daaf9c P 9e91172c416f417386d7a002c121ca48: No bootstrap required, opened a new log
I20260929 17:53:11.437183 2740 ts_tablet_manager.cc:1397] T c477d231a7ce4d61b3b472e027daaf9c P 9e91172c416f417386d7a002c121ca48: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260929 17:53:11.437134 2741 consensus_queue.cc:260] T c477d231a7ce4d61b3b472e027daaf9c P 3cddbb0910424ec2b410a25791b1ba46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e91172c416f417386d7a002c121ca48" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37663 } } peers { permanent_uuid: "b367e800a8fb4f0aa965712d54ee86f5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46719 } } peers { permanent_uuid: "3cddbb0910424ec2b410a25791b1ba46" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37811 } }
I20260929 17:53:11.437409 2741 ts_tablet_manager.cc:1428] T c477d231a7ce4d61b3b472e027daaf9c P 3cddbb0910424ec2b410a25791b1ba46: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260929 17:53:11.437526 2740 raft_consensus.cc:348] T c477d231a7ce4d61b3b472e027daaf9c P 9e91172c416f417386d7a002c121ca48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e91172c416f417386d7a002c121ca48" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37663 } } peers { permanent_uuid: "b367e800a8fb4f0aa965712d54ee86f5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46719 } } peers { permanent_uuid: "3cddbb0910424ec2b410a25791b1ba46" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37811 } }
I20260929 17:53:11.437618 2740 raft_consensus.cc:374] T c477d231a7ce4d61b3b472e027daaf9c P 9e91172c416f417386d7a002c121ca48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:53:11.437654 2740 raft_consensus.cc:729] T c477d231a7ce4d61b3b472e027daaf9c P 9e91172c416f417386d7a002c121ca48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e91172c416f417386d7a002c121ca48, State: Initialized, Role: FOLLOWER
I20260929 17:53:11.437767 2740 consensus_queue.cc:260] T c477d231a7ce4d61b3b472e027daaf9c P 9e91172c416f417386d7a002c121ca48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e91172c416f417386d7a002c121ca48" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37663 } } peers { permanent_uuid: "b367e800a8fb4f0aa965712d54ee86f5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46719 } } peers { permanent_uuid: "3cddbb0910424ec2b410a25791b1ba46" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37811 } }
I20260929 17:53:11.437968 2740 ts_tablet_manager.cc:1428] T c477d231a7ce4d61b3b472e027daaf9c P 9e91172c416f417386d7a002c121ca48: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260929 17:53:11.439847 2739 tablet_bootstrap.cc:492] T c477d231a7ce4d61b3b472e027daaf9c P b367e800a8fb4f0aa965712d54ee86f5: Bootstrap starting.
I20260929 17:53:11.440560 2739 tablet_bootstrap.cc:654] T c477d231a7ce4d61b3b472e027daaf9c P b367e800a8fb4f0aa965712d54ee86f5: Neither blocks nor log segments found. Creating new log.
I20260929 17:53:11.441504 2739 tablet_bootstrap.cc:492] T c477d231a7ce4d61b3b472e027daaf9c P b367e800a8fb4f0aa965712d54ee86f5: No bootstrap required, opened a new log
I20260929 17:53:11.441577 2739 ts_tablet_manager.cc:1397] T c477d231a7ce4d61b3b472e027daaf9c P b367e800a8fb4f0aa965712d54ee86f5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 17:53:11.441941 2739 raft_consensus.cc:348] T c477d231a7ce4d61b3b472e027daaf9c P b367e800a8fb4f0aa965712d54ee86f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e91172c416f417386d7a002c121ca48" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37663 } } peers { permanent_uuid: "b367e800a8fb4f0aa965712d54ee86f5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46719 } } peers { permanent_uuid: "3cddbb0910424ec2b410a25791b1ba46" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37811 } }
I20260929 17:53:11.442050 2739 raft_consensus.cc:374] T c477d231a7ce4d61b3b472e027daaf9c P b367e800a8fb4f0aa965712d54ee86f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:53:11.442083 2739 raft_consensus.cc:729] T c477d231a7ce4d61b3b472e027daaf9c P b367e800a8fb4f0aa965712d54ee86f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b367e800a8fb4f0aa965712d54ee86f5, State: Initialized, Role: FOLLOWER
I20260929 17:53:11.442209 2739 consensus_queue.cc:260] T c477d231a7ce4d61b3b472e027daaf9c P b367e800a8fb4f0aa965712d54ee86f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e91172c416f417386d7a002c121ca48" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37663 } } peers { permanent_uuid: "b367e800a8fb4f0aa965712d54ee86f5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46719 } } peers { permanent_uuid: "3cddbb0910424ec2b410a25791b1ba46" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37811 } }
I20260929 17:53:11.442432 2739 ts_tablet_manager.cc:1428] T c477d231a7ce4d61b3b472e027daaf9c P b367e800a8fb4f0aa965712d54ee86f5: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260929 17:53:11.667547 2746 raft_consensus.cc:482] T c477d231a7ce4d61b3b472e027daaf9c P 3cddbb0910424ec2b410a25791b1ba46 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 17:53:11.667691 2746 raft_consensus.cc:504] T c477d231a7ce4d61b3b472e027daaf9c P 3cddbb0910424ec2b410a25791b1ba46 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e91172c416f417386d7a002c121ca48" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37663 } } peers { permanent_uuid: "b367e800a8fb4f0aa965712d54ee86f5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46719 } } peers { permanent_uuid: "3cddbb0910424ec2b410a25791b1ba46" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37811 } }
I20260929 17:53:11.668799 2746 leader_election.cc:288] T c477d231a7ce4d61b3b472e027daaf9c P 3cddbb0910424ec2b410a25791b1ba46 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9e91172c416f417386d7a002c121ca48 (127.0.98.1:37663), b367e800a8fb4f0aa965712d54ee86f5 (127.0.98.2:46719)
I20260929 17:53:11.673259 2749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:11.669643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53667 (local address 127.0.98.2:46719)
0929 17:53:11.669762 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:11.669765 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:11.669889 (+ 124us) server_negotiation.cc:407] Connection header received
0929 17:53:11.670004 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:11.670007 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:11.670050 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:11.670115 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:53:11.670409 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:11.671136 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:11.672259 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:11.672706 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:11.672806 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:11.672869 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0929 17:53:11.673057 (+ 188us) server_negotiation.cc:299] Negotiation successful
0929 17:53:11.673129 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260929 17:53:11.673264 2759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:11.669496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46719 (local address 127.0.98.3:53667)
0929 17:53:11.669818 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:11.669830 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 17:53:11.669921 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:53:11.670131 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:53:11.670134 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:53:11.670149 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:53:11.670302 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:11.670309 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:11.671249 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:11.671253 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:11.672138 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:11.672147 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:11.672809 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:11.672812 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:11.672904 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:11.672948 (+ 44us) client_negotiation.cc:769] Sending connection context
0929 17:53:11.673043 (+ 95us) client_negotiation.cc:240] Negotiation successful
0929 17:53:11.673113 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"mutex_wait_us":24,"thread_start_us":83,"threads_started":1}
I20260929 17:53:11.673681 2757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:11.668990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37663 (local address 127.0.98.3:56491)
0929 17:53:11.669370 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:11.669384 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 17:53:11.669487 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:53:11.670024 (+ 537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:53:11.670028 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:53:11.670046 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:53:11.670194 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:11.670202 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:11.671207 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:11.671210 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:11.672470 (+ 1260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:11.672479 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:11.673332 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:11.673336 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:11.673431 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:11.673448 (+ 17us) client_negotiation.cc:769] Sending connection context
0929 17:53:11.673498 (+ 50us) client_negotiation.cc:240] Negotiation successful
0929 17:53:11.673554 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":312,"mutex_wait_us":30,"thread_start_us":97,"threads_started":1}
I20260929 17:53:11.673790 2440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c477d231a7ce4d61b3b472e027daaf9c" candidate_uuid: "3cddbb0910424ec2b410a25791b1ba46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b367e800a8fb4f0aa965712d54ee86f5" is_pre_election: true
I20260929 17:53:11.673873 2758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:11.669141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56491 (local address 127.0.98.1:37663)
0929 17:53:11.669362 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:11.669378 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:11.669573 (+ 195us) server_negotiation.cc:407] Connection header received
0929 17:53:11.669629 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:11.669631 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:11.669796 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:11.669873 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:53:11.670310 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:11.671105 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:11.672588 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:11.673193 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:11.673298 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:11.673645 (+ 347us) server_negotiation.cc:1035] Waiting for connection context
0929 17:53:11.673678 (+ 33us) server_negotiation.cc:299] Negotiation successful
0929 17:53:11.673761 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":63,"threads_started":1}
I20260929 17:53:11.673935 2440 raft_consensus.cc:2454] T c477d231a7ce4d61b3b472e027daaf9c P b367e800a8fb4f0aa965712d54ee86f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3cddbb0910424ec2b410a25791b1ba46 in term 0.
I20260929 17:53:11.674141 2745 raft_consensus.cc:482] T c477d231a7ce4d61b3b472e027daaf9c P 9e91172c416f417386d7a002c121ca48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 17:53:11.674237 2745 raft_consensus.cc:504] T c477d231a7ce4d61b3b472e027daaf9c P 9e91172c416f417386d7a002c121ca48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e91172c416f417386d7a002c121ca48" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37663 } } peers { permanent_uuid: "b367e800a8fb4f0aa965712d54ee86f5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46719 } } peers { permanent_uuid: "3cddbb0910424ec2b410a25791b1ba46" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37811 } }
I20260929 17:53:11.674300 2511 leader_election.cc:302] T c477d231a7ce4d61b3b472e027daaf9c P 3cddbb0910424ec2b410a25791b1ba46 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3cddbb0910424ec2b410a25791b1ba46, b367e800a8fb4f0aa965712d54ee86f5; no voters:
I20260929 17:53:11.674337 2300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c477d231a7ce4d61b3b472e027daaf9c" candidate_uuid: "3cddbb0910424ec2b410a25791b1ba46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e91172c416f417386d7a002c121ca48" is_pre_election: true
I20260929 17:53:11.674458 2300 raft_consensus.cc:2454] T c477d231a7ce4d61b3b472e027daaf9c P 9e91172c416f417386d7a002c121ca48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3cddbb0910424ec2b410a25791b1ba46 in term 0.
I20260929 17:53:11.674531 2746 raft_consensus.cc:2781] T c477d231a7ce4d61b3b472e027daaf9c P 3cddbb0910424ec2b410a25791b1ba46 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 17:53:11.674578 2746 raft_consensus.cc:482] T c477d231a7ce4d61b3b472e027daaf9c P 3cddbb0910424ec2b410a25791b1ba46 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 17:53:11.674612 2746 raft_consensus.cc:3037] T c477d231a7ce4d61b3b472e027daaf9c P 3cddbb0910424ec2b410a25791b1ba46 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:53:11.674603 2745 leader_election.cc:288] T c477d231a7ce4d61b3b472e027daaf9c P 9e91172c416f417386d7a002c121ca48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b367e800a8fb4f0aa965712d54ee86f5 (127.0.98.2:46719), 3cddbb0910424ec2b410a25791b1ba46 (127.0.98.3:37811)
I20260929 17:53:11.674921 2440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c477d231a7ce4d61b3b472e027daaf9c" candidate_uuid: "9e91172c416f417386d7a002c121ca48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b367e800a8fb4f0aa965712d54ee86f5" is_pre_election: true
I20260929 17:53:11.675022 2440 raft_consensus.cc:2454] T c477d231a7ce4d61b3b472e027daaf9c P b367e800a8fb4f0aa965712d54ee86f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9e91172c416f417386d7a002c121ca48 in term 0.
I20260929 17:53:11.675043 2580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c477d231a7ce4d61b3b472e027daaf9c" candidate_uuid: "9e91172c416f417386d7a002c121ca48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3cddbb0910424ec2b410a25791b1ba46" is_pre_election: true
I20260929 17:53:11.675271 2231 leader_election.cc:302] T c477d231a7ce4d61b3b472e027daaf9c P 9e91172c416f417386d7a002c121ca48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9e91172c416f417386d7a002c121ca48, b367e800a8fb4f0aa965712d54ee86f5; no voters:
I20260929 17:53:11.675413 2745 raft_consensus.cc:2781] T c477d231a7ce4d61b3b472e027daaf9c P 9e91172c416f417386d7a002c121ca48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 17:53:11.675463 2745 raft_consensus.cc:482] T c477d231a7ce4d61b3b472e027daaf9c P 9e91172c416f417386d7a002c121ca48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 17:53:11.675495 2745 raft_consensus.cc:3037] T c477d231a7ce4d61b3b472e027daaf9c P 9e91172c416f417386d7a002c121ca48 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:53:11.675727 2746 raft_consensus.cc:504] T c477d231a7ce4d61b3b472e027daaf9c P 3cddbb0910424ec2b410a25791b1ba46 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e91172c416f417386d7a002c121ca48" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37663 } } peers { permanent_uuid: "b367e800a8fb4f0aa965712d54ee86f5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46719 } } peers { permanent_uuid: "3cddbb0910424ec2b410a25791b1ba46" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37811 } }
I20260929 17:53:11.675940 2580 raft_consensus.cc:2379] T c477d231a7ce4d61b3b472e027daaf9c P 3cddbb0910424ec2b410a25791b1ba46 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9e91172c416f417386d7a002c121ca48 in current term 1: Already voted for candidate 3cddbb0910424ec2b410a25791b1ba46 in this term.
I20260929 17:53:11.676028 2746 leader_election.cc:288] T c477d231a7ce4d61b3b472e027daaf9c P 3cddbb0910424ec2b410a25791b1ba46 [CANDIDATE]: Term 1 election: Requested vote from peers 9e91172c416f417386d7a002c121ca48 (127.0.98.1:37663), b367e800a8fb4f0aa965712d54ee86f5 (127.0.98.2:46719)
I20260929 17:53:11.676348 2300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c477d231a7ce4d61b3b472e027daaf9c" candidate_uuid: "3cddbb0910424ec2b410a25791b1ba46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e91172c416f417386d7a002c121ca48"
I20260929 17:53:11.676545 2440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c477d231a7ce4d61b3b472e027daaf9c" candidate_uuid: "3cddbb0910424ec2b410a25791b1ba46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b367e800a8fb4f0aa965712d54ee86f5"
I20260929 17:53:11.676568 2745 raft_consensus.cc:504] T c477d231a7ce4d61b3b472e027daaf9c P 9e91172c416f417386d7a002c121ca48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e91172c416f417386d7a002c121ca48" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37663 } } peers { permanent_uuid: "b367e800a8fb4f0aa965712d54ee86f5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46719 } } peers { permanent_uuid: "3cddbb0910424ec2b410a25791b1ba46" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37811 } }
I20260929 17:53:11.676628 2440 raft_consensus.cc:3037] T c477d231a7ce4d61b3b472e027daaf9c P b367e800a8fb4f0aa965712d54ee86f5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:53:11.676792 2300 raft_consensus.cc:2379] T c477d231a7ce4d61b3b472e027daaf9c P 9e91172c416f417386d7a002c121ca48 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 3cddbb0910424ec2b410a25791b1ba46 in current term 1: Already voted for candidate 9e91172c416f417386d7a002c121ca48 in this term.
I20260929 17:53:11.676858 2745 leader_election.cc:288] T c477d231a7ce4d61b3b472e027daaf9c P 9e91172c416f417386d7a002c121ca48 [CANDIDATE]: Term 1 election: Requested vote from peers b367e800a8fb4f0aa965712d54ee86f5 (127.0.98.2:46719), 3cddbb0910424ec2b410a25791b1ba46 (127.0.98.3:37811)
I20260929 17:53:11.677204 2580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c477d231a7ce4d61b3b472e027daaf9c" candidate_uuid: "9e91172c416f417386d7a002c121ca48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3cddbb0910424ec2b410a25791b1ba46"
I20260929 17:53:11.677320 2580 raft_consensus.cc:2379] T c477d231a7ce4d61b3b472e027daaf9c P 3cddbb0910424ec2b410a25791b1ba46 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 9e91172c416f417386d7a002c121ca48 in current term 1: Already voted for candidate 3cddbb0910424ec2b410a25791b1ba46 in this term.
I20260929 17:53:11.677690 2440 raft_consensus.cc:2454] T c477d231a7ce4d61b3b472e027daaf9c P b367e800a8fb4f0aa965712d54ee86f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3cddbb0910424ec2b410a25791b1ba46 in term 1.
I20260929 17:53:11.677919 2511 leader_election.cc:302] T c477d231a7ce4d61b3b472e027daaf9c P 3cddbb0910424ec2b410a25791b1ba46 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3cddbb0910424ec2b410a25791b1ba46, b367e800a8fb4f0aa965712d54ee86f5; no voters:
I20260929 17:53:11.678109 2440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c477d231a7ce4d61b3b472e027daaf9c" candidate_uuid: "9e91172c416f417386d7a002c121ca48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b367e800a8fb4f0aa965712d54ee86f5"
I20260929 17:53:11.678277 2440 raft_consensus.cc:2379] T c477d231a7ce4d61b3b472e027daaf9c P b367e800a8fb4f0aa965712d54ee86f5 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 9e91172c416f417386d7a002c121ca48 in current term 1: Already voted for candidate 3cddbb0910424ec2b410a25791b1ba46 in this term.
I20260929 17:53:11.678263 2746 raft_consensus.cc:2781] T c477d231a7ce4d61b3b472e027daaf9c P 3cddbb0910424ec2b410a25791b1ba46 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 17:53:11.678457 2746 raft_consensus.cc:686] T c477d231a7ce4d61b3b472e027daaf9c P 3cddbb0910424ec2b410a25791b1ba46 [term 1 LEADER]: Becoming Leader. State: Replica: 3cddbb0910424ec2b410a25791b1ba46, State: Running, Role: LEADER
I20260929 17:53:11.678529 2231 leader_election.cc:302] T c477d231a7ce4d61b3b472e027daaf9c P 9e91172c416f417386d7a002c121ca48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9e91172c416f417386d7a002c121ca48; no voters: 3cddbb0910424ec2b410a25791b1ba46, b367e800a8fb4f0aa965712d54ee86f5
I20260929 17:53:11.678681 2745 raft_consensus.cc:2726] T c477d231a7ce4d61b3b472e027daaf9c P 9e91172c416f417386d7a002c121ca48 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260929 17:53:11.678692 2746 consensus_queue.cc:237] T c477d231a7ce4d61b3b472e027daaf9c P 3cddbb0910424ec2b410a25791b1ba46 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e91172c416f417386d7a002c121ca48" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37663 } } peers { permanent_uuid: "b367e800a8fb4f0aa965712d54ee86f5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46719 } } peers { permanent_uuid: "3cddbb0910424ec2b410a25791b1ba46" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37811 } }
I20260929 17:53:11.680732 2143 catalog_manager.cc:5495] T c477d231a7ce4d61b3b472e027daaf9c P 3cddbb0910424ec2b410a25791b1ba46 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3cddbb0910424ec2b410a25791b1ba46 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "3cddbb0910424ec2b410a25791b1ba46" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e91172c416f417386d7a002c121ca48" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37663 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b367e800a8fb4f0aa965712d54ee86f5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46719 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3cddbb0910424ec2b410a25791b1ba46" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37811 } health_report { overall_health: HEALTHY } } }
I20260929 17:53:11.737394 2762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:11.733915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43050 (local address 127.0.98.3:37811)
0929 17:53:11.734180 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:11.734186 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:11.734197 (+ 11us) server_negotiation.cc:407] Connection header received
0929 17:53:11.734274 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:11.734276 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:11.734328 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:11.734424 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 17:53:11.734636 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:11.735196 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:11.736075 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:11.736334 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:11.736418 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:11.736605 (+ 187us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 17:53:11.737100 (+ 495us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 17:53:11.737180 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0929 17:53:11.737221 (+ 41us) server_negotiation.cc:299] Negotiation successful
0929 17:53:11.737267 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":204,"thread_start_us":117,"threads_started":1}
I20260929 17:53:11.746806 2749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:11.743968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33497 (local address 127.0.98.2:46719)
0929 17:53:11.744074 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:11.744077 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:11.744215 (+ 138us) server_negotiation.cc:407] Connection header received
0929 17:53:11.744314 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:11.744316 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:11.744359 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:11.744418 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 17:53:11.744611 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:11.745134 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:11.745915 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:11.746117 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:11.746184 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:11.746409 (+ 225us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 17:53:11.746473 (+ 64us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 17:53:11.746519 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0929 17:53:11.746649 (+ 130us) server_negotiation.cc:299] Negotiation successful
0929 17:53:11.746691 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260929 17:53:11.747220 2765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:11.743873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46719 (local address 127.0.98.3:33497)
0929 17:53:11.744147 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:11.744159 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 17:53:11.744242 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:53:11.744427 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:53:11.744429 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:53:11.744435 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 17:53:11.744527 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:11.744531 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:11.745211 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:11.745213 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:11.745832 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:11.745839 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:11.746190 (+ 351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:11.746192 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:11.746273 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:11.746313 (+ 40us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 17:53:11.746562 (+ 249us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 17:53:11.746570 (+ 8us) client_negotiation.cc:769] Sending connection context
0929 17:53:11.746640 (+ 70us) client_negotiation.cc:240] Negotiation successful
0929 17:53:11.746712 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":85,"threads_started":1}
I20260929 17:53:11.752892 2300 raft_consensus.cc:1261] T c477d231a7ce4d61b3b472e027daaf9c P 9e91172c416f417386d7a002c121ca48 [term 1 FOLLOWER]: Refusing update from remote peer 3cddbb0910424ec2b410a25791b1ba46: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 17:53:11.752952 2440 raft_consensus.cc:1261] T c477d231a7ce4d61b3b472e027daaf9c P b367e800a8fb4f0aa965712d54ee86f5 [term 1 FOLLOWER]: Refusing update from remote peer 3cddbb0910424ec2b410a25791b1ba46: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 17:53:11.753456 2760 consensus_queue.cc:1035] T c477d231a7ce4d61b3b472e027daaf9c P 3cddbb0910424ec2b410a25791b1ba46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e91172c416f417386d7a002c121ca48" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37663 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 17:53:11.753645 2746 consensus_queue.cc:1035] T c477d231a7ce4d61b3b472e027daaf9c P 3cddbb0910424ec2b410a25791b1ba46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b367e800a8fb4f0aa965712d54ee86f5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 17:53:11.769922 2770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:11.766450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37811 (local address 127.0.98.2:50113)
0929 17:53:11.766919 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:11.766934 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 17:53:11.767036 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:53:11.767258 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:53:11.767261 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:53:11.767269 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 17:53:11.767367 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:11.767373 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:11.768045 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:11.768049 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:11.768765 (+ 716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:11.768773 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:11.769301 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:11.769304 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:11.769388 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:11.769417 (+ 29us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 17:53:11.769653 (+ 236us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 17:53:11.769660 (+ 7us) client_negotiation.cc:769] Sending connection context
0929 17:53:11.769723 (+ 63us) client_negotiation.cc:240] Negotiation successful
0929 17:53:11.769799 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":104,"threads_started":1}
I20260929 17:53:11.769912 2762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:11.766762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50113 (local address 127.0.98.3:37811)
0929 17:53:11.766883 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:11.766888 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:11.767006 (+ 118us) server_negotiation.cc:407] Connection header received
0929 17:53:11.767120 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:11.767123 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:11.767180 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:11.767246 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 17:53:11.767457 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:11.767952 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:11.768865 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:11.769215 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:11.769305 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:11.769518 (+ 213us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 17:53:11.769586 (+ 68us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 17:53:11.769639 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0929 17:53:11.769732 (+ 93us) server_negotiation.cc:299] Negotiation successful
0929 17:53:11.769782 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
W20260929 17:53:11.770612 2713 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260929 17:53:13.329023 2143 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57484:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 17:53:13.329227 2143 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57484:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 17:53:13.331594 2143 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 72f917d01d69473394db09e8e813365f: Sending DeleteTablet for 3 replicas of tablet c477d231a7ce4d61b3b472e027daaf9c
I20260929 17:53:13.332178 2560 tablet_service.cc:1514] Processing DeleteTablet for tablet c477d231a7ce4d61b3b472e027daaf9c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 17:53:13 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43038
I20260929 17:53:13.332171 2280 tablet_service.cc:1514] Processing DeleteTablet for tablet c477d231a7ce4d61b3b472e027daaf9c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 17:53:13 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59114
I20260929 17:53:13.332213 2420 tablet_service.cc:1514] Processing DeleteTablet for tablet c477d231a7ce4d61b3b472e027daaf9c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 17:53:13 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54000
I20260929 17:53:13.332584 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskup3Soe/build/debug/bin/kudu with pid 2215
I20260929 17:53:13.332602 2794 tablet_replica.cc:330] T c477d231a7ce4d61b3b472e027daaf9c P 9e91172c416f417386d7a002c121ca48: stopping tablet replica
I20260929 17:53:13.333235 2795 tablet_replica.cc:330] T c477d231a7ce4d61b3b472e027daaf9c P b367e800a8fb4f0aa965712d54ee86f5: stopping tablet replica
I20260929 17:53:13.333365 2795 raft_consensus.cc:2229] T c477d231a7ce4d61b3b472e027daaf9c P b367e800a8fb4f0aa965712d54ee86f5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 17:53:13.333472 2795 raft_consensus.cc:2258] T c477d231a7ce4d61b3b472e027daaf9c P b367e800a8fb4f0aa965712d54ee86f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 17:53:13.333709 2793 tablet_replica.cc:330] T c477d231a7ce4d61b3b472e027daaf9c P 3cddbb0910424ec2b410a25791b1ba46: stopping tablet replica
I20260929 17:53:13.333827 2793 raft_consensus.cc:2229] T c477d231a7ce4d61b3b472e027daaf9c P 3cddbb0910424ec2b410a25791b1ba46 [term 1 LEADER]: Raft consensus shutting down.
I20260929 17:53:13.334112 2793 raft_consensus.cc:2258] T c477d231a7ce4d61b3b472e027daaf9c P 3cddbb0910424ec2b410a25791b1ba46 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 17:53:13.334887 2795 ts_tablet_manager.cc:1905] T c477d231a7ce4d61b3b472e027daaf9c P b367e800a8fb4f0aa965712d54ee86f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 17:53:13.337430 2793 ts_tablet_manager.cc:1905] T c477d231a7ce4d61b3b472e027daaf9c P 3cddbb0910424ec2b410a25791b1ba46: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 17:53:13.337852 2795 ts_tablet_manager.cc:1918] T c477d231a7ce4d61b3b472e027daaf9c P b367e800a8fb4f0aa965712d54ee86f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 17:53:13.337916 2795 log.cc:1196] T c477d231a7ce4d61b3b472e027daaf9c P b367e800a8fb4f0aa965712d54ee86f5: Deleting WAL directory at /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704375975685-392-0/minicluster-data/ts-1/wal/wals/c477d231a7ce4d61b3b472e027daaf9c
I20260929 17:53:13.338142 2795 ts_tablet_manager.cc:1939] T c477d231a7ce4d61b3b472e027daaf9c P b367e800a8fb4f0aa965712d54ee86f5: Deleting consensus metadata
I20260929 17:53:13.338977 2127 catalog_manager.cc:4841] TS b367e800a8fb4f0aa965712d54ee86f5 (127.0.98.2:46719): tablet c477d231a7ce4d61b3b472e027daaf9c (table test-table [id=82b1a4883674434eb79d058736614d79]) successfully deleted
I20260929 17:53:13.339486 2793 ts_tablet_manager.cc:1918] T c477d231a7ce4d61b3b472e027daaf9c P 3cddbb0910424ec2b410a25791b1ba46: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 17:53:13.339551 2793 log.cc:1196] T c477d231a7ce4d61b3b472e027daaf9c P 3cddbb0910424ec2b410a25791b1ba46: Deleting WAL directory at /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790704375975685-392-0/minicluster-data/ts-2/wal/wals/c477d231a7ce4d61b3b472e027daaf9c
I20260929 17:53:13.339787 2793 ts_tablet_manager.cc:1939] T c477d231a7ce4d61b3b472e027daaf9c P 3cddbb0910424ec2b410a25791b1ba46: Deleting consensus metadata
W20260929 17:53:13.340283 2127 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:37663 (error 108)
I20260929 17:53:13.340372 2128 catalog_manager.cc:4841] TS 3cddbb0910424ec2b410a25791b1ba46 (127.0.98.3:37811): tablet c477d231a7ce4d61b3b472e027daaf9c (table test-table [id=82b1a4883674434eb79d058736614d79]) successfully deleted
I20260929 17:53:13.340981 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskup3Soe/build/debug/bin/kudu with pid 2355
I20260929 17:53:13.342037 2797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:13.340938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37663 (local address 127.0.0.1:59124)
0929 17:53:13.341848 (+ 910us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:13.341938 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":832,"thread_start_us":89,"threads_started":1}
W20260929 17:53:13.342157 2127 catalog_manager.cc:4584] TS 9e91172c416f417386d7a002c121ca48 (127.0.98.1:37663): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c477d231a7ce4d61b3b472e027daaf9c: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37663: connect: Connection refused (error 111)
I20260929 17:53:13.347723 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskup3Soe/build/debug/bin/kudu with pid 2495
I20260929 17:53:13.353327 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskup3Soe/build/debug/bin/kudu with pid 2111
2026-09-29T17:53:13Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6202 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704375975685-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 17:53:13 dist-test-slave-53nh krb5kdc[2800](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 17:53:13 dist-test-slave-53nh krb5kdc[2800](info): set up 2 sockets
Sep 29 17:53:13 dist-test-slave-53nh krb5kdc[2800](info): commencing operation
krb5kdc: starting...
W20260929 17:53:14.304690 392 mini_kdc.cc:121] Time spent starting KDC: real 0.930s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704375975685-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704375975685-392-0/krb5kdc/test-user.keytab.
Sep 29 17:53:14 dist-test-slave-53nh krb5kdc[2800](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704394, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T17:53:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T17:53:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704375975685-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704375975685-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 17:53:14.417768 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskup3Soe/build/debug/bin/kudu
/tmp/dist-test-taskup3Soe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704375975685-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704375975685-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704375975685-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704375975685-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38059
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704375975685-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38059
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704375975685-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704375975685-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704375975685-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:53:14.525427 2813 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:53:14.525725 2813 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:53:14.525776 2813 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:53:14.530061 2813 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 17:53:14.530136 2813 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:53:14.530155 2813 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:53:14.530170 2813 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 17:53:14.530184 2813 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 17:53:14.535615 2813 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704375975685-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704375975685-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38059
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704375975685-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38059
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704375975685-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2813
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704375975685-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:53:14.536626 2813 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:53:14.537636 2813 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:53:14.545542 2821 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:53:14.545555 2818 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:53:14.545572 2819 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:53:14.545867 2813 server_base.cc:1034] running on GCE node
I20260929 17:53:14.546407 2813 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:53:14.547473 2813 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:53:14.548629 2813 hybrid_clock.cc:648] HybridClock initialized: now 1790704394548605 us; error 38 us; skew 500 ppm
Sep 29 17:53:14 dist-test-slave-53nh krb5kdc[2800](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704394, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:53:14.551399 2813 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 17:53:14.552270 2813 webserver.cc:460] Webserver started at http://127.0.98.62:32969/ using document root <none> and password file <none>
I20260929 17:53:14.552811 2813 fs_manager.cc:359] Metadata directory not provided
I20260929 17:53:14.552876 2813 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:53:14.553105 2813 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:53:14.555097 2813 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704375975685-392-0/minicluster-data/master-0/data/instance:
uuid: "858feb59638648a0abc454251983153a"
format_stamp: "Formatted at 2026-09-29 17:53:14 on dist-test-slave-53nh"
server_key: "2c0b3af42d923ec68b688c8cb77e5ff2"
server_key_iv: "2b5497d2ba231b9a2c4e991ec9a6c84d"
server_key_version: "encryptionkey@0"
I20260929 17:53:14.555627 2813 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704375975685-392-0/minicluster-data/master-0/wal/instance:
uuid: "858feb59638648a0abc454251983153a"
format_stamp: "Formatted at 2026-09-29 17:53:14 on dist-test-slave-53nh"
server_key: "2c0b3af42d923ec68b688c8cb77e5ff2"
server_key_iv: "2b5497d2ba231b9a2c4e991ec9a6c84d"
server_key_version: "encryptionkey@0"
I20260929 17:53:14.559151 2813 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 17:53:14.561498 2828 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:53:14.562724 2813 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 17:53:14.562832 2813 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704375975685-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704375975685-392-0/minicluster-data/master-0/wal
uuid: "858feb59638648a0abc454251983153a"
format_stamp: "Formatted at 2026-09-29 17:53:14 on dist-test-slave-53nh"
server_key: "2c0b3af42d923ec68b688c8cb77e5ff2"
server_key_iv: "2b5497d2ba231b9a2c4e991ec9a6c84d"
server_key_version: "encryptionkey@0"
I20260929 17:53:14.562924 2813 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704375975685-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704375975685-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704375975685-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:53:14.582849 2813 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:53:14.585626 2813 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:53:14.585794 2813 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:53:14.592173 2813 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38059
I20260929 17:53:14.592200 2880 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38059 every 8 connection(s)
I20260929 17:53:14.593070 2813 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704375975685-392-0/minicluster-data/master-0/data/info.pb
I20260929 17:53:14.593408 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskup3Soe/build/debug/bin/kudu as pid 2813
I20260929 17:53:14.593516 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704375975685-392-0/minicluster-data/master-0/wal/instance
I20260929 17:53:14.593814 392 external_mini_cluster.cc:1428] Setting key 062110de07b814eca142a6a69d5475d8
I20260929 17:53:14.596026 2881 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:53:14.602203 2881 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 858feb59638648a0abc454251983153a: Bootstrap starting.
Sep 29 17:53:14 dist-test-slave-53nh krb5kdc[2800](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704394, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 17:53:14.604285 2881 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 858feb59638648a0abc454251983153a: Neither blocks nor log segments found. Creating new log.
I20260929 17:53:14.604948 2881 log.cc:824] T 00000000000000000000000000000000 P 858feb59638648a0abc454251983153a: Log is configured to *not* fsync() on all Append() calls
I20260929 17:53:14.606729 2881 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 858feb59638648a0abc454251983153a: No bootstrap required, opened a new log
I20260929 17:53:14.606957 2884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:14.595170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54120 (local address 127.0.98.62:38059)
0929 17:53:14.595560 (+ 390us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:14.595575 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:14.595610 (+ 35us) server_negotiation.cc:407] Connection header received
0929 17:53:14.596240 (+ 630us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:14.596258 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:14.596584 (+ 326us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:14.596856 (+ 272us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:53:14.597728 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:14.598686 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:14.599799 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:14.600234 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:14.600362 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:14.602898 (+ 2536us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:53:14.602902 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:53:14.602915 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:53:14.602941 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:53:14.604455 (+ 1514us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:53:14.604831 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:53:14.604836 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:53:14.604840 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:53:14.604940 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:53:14.605209 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:53:14.605212 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:53:14.605214 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:53:14.605574 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:53:14.605736 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0929 17:53:14.606009 (+ 273us) server_negotiation.cc:299] Negotiation successful
0929 17:53:14.606257 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":259,"thread_start_us":120,"threads_started":1}
I20260929 17:53:14.610383 2881 raft_consensus.cc:348] T 00000000000000000000000000000000 P 858feb59638648a0abc454251983153a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "858feb59638648a0abc454251983153a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38059 } }
I20260929 17:53:14.610591 2881 raft_consensus.cc:374] T 00000000000000000000000000000000 P 858feb59638648a0abc454251983153a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:53:14.610646 2881 raft_consensus.cc:729] T 00000000000000000000000000000000 P 858feb59638648a0abc454251983153a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 858feb59638648a0abc454251983153a, State: Initialized, Role: FOLLOWER
I20260929 17:53:14.611029 2881 consensus_queue.cc:260] T 00000000000000000000000000000000 P 858feb59638648a0abc454251983153a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "858feb59638648a0abc454251983153a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38059 } }
I20260929 17:53:14.611150 2881 raft_consensus.cc:388] T 00000000000000000000000000000000 P 858feb59638648a0abc454251983153a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 17:53:14.611210 2881 raft_consensus.cc:482] T 00000000000000000000000000000000 P 858feb59638648a0abc454251983153a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 17:53:14.611301 2881 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 858feb59638648a0abc454251983153a [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:53:14.612109 2881 raft_consensus.cc:504] T 00000000000000000000000000000000 P 858feb59638648a0abc454251983153a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "858feb59638648a0abc454251983153a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38059 } }
I20260929 17:53:14.612406 2881 leader_election.cc:302] T 00000000000000000000000000000000 P 858feb59638648a0abc454251983153a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 858feb59638648a0abc454251983153a; no voters:
I20260929 17:53:14.612670 2881 leader_election.cc:288] T 00000000000000000000000000000000 P 858feb59638648a0abc454251983153a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 17:53:14.612777 2886 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 858feb59638648a0abc454251983153a [term 1 FOLLOWER]: Leader election won for term 1
I20260929 17:53:14.613031 2886 raft_consensus.cc:686] T 00000000000000000000000000000000 P 858feb59638648a0abc454251983153a [term 1 LEADER]: Becoming Leader. State: Replica: 858feb59638648a0abc454251983153a, State: Running, Role: LEADER
I20260929 17:53:14.613315 2886 consensus_queue.cc:237] T 00000000000000000000000000000000 P 858feb59638648a0abc454251983153a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "858feb59638648a0abc454251983153a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38059 } }
I20260929 17:53:14.613821 2881 sys_catalog.cc:564] T 00000000000000000000000000000000 P 858feb59638648a0abc454251983153a [sys.catalog]: configured and running, proceeding with master startup.
I20260929 17:53:14.614812 2888 sys_catalog.cc:455] T 00000000000000000000000000000000 P 858feb59638648a0abc454251983153a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 858feb59638648a0abc454251983153a. Latest consensus state: current_term: 1 leader_uuid: "858feb59638648a0abc454251983153a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "858feb59638648a0abc454251983153a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38059 } } }
I20260929 17:53:14.614929 2888 sys_catalog.cc:458] T 00000000000000000000000000000000 P 858feb59638648a0abc454251983153a [sys.catalog]: This master's current role is: LEADER
I20260929 17:53:14.614988 2887 sys_catalog.cc:455] T 00000000000000000000000000000000 P 858feb59638648a0abc454251983153a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "858feb59638648a0abc454251983153a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "858feb59638648a0abc454251983153a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38059 } } }
I20260929 17:53:14.615077 2887 sys_catalog.cc:458] T 00000000000000000000000000000000 P 858feb59638648a0abc454251983153a [sys.catalog]: This master's current role is: LEADER
I20260929 17:53:14.616076 2895 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 17:53:14.618588 2895 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 17:53:14.623711 2895 catalog_manager.cc:1347] Generated new cluster ID: b306b814935745aa8c3c35753b75c8ea
I20260929 17:53:14.623765 2895 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 17:53:14.634505 2895 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 17:53:14.635840 2895 catalog_manager.cc:1504] Loading token signing keys...
I20260929 17:53:14.641119 2895 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 858feb59638648a0abc454251983153a: Generated new TSK 0
I20260929 17:53:14.641794 2895 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704375975685-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704375975685-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260929 17:53:14.668095 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskup3Soe/build/debug/bin/kudu
/tmp/dist-test-taskup3Soe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704375975685-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704375975685-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704375975685-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704375975685-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38059
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704375975685-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704375975685-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704375975685-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704375975685-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:53:14.775039 2907 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:53:14.775300 2907 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:53:14.775358 2907 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:53:14.779352 2907 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:53:14.779412 2907 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:53:14.779491 2907 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 17:53:14.784584 2907 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704375975685-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704375975685-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704375975685-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704375975685-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38059
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2907
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704375975685-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:53:14.785599 2907 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:53:14.786558 2907 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:53:14.795451 2915 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:53:14.795387 2913 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:53:14.795380 2912 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:53:14.795807 2907 server_base.cc:1034] running on GCE node
I20260929 17:53:14.796183 2907 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:53:14.796803 2907 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:53:14.797988 2907 hybrid_clock.cc:648] HybridClock initialized: now 1790704394797962 us; error 32 us; skew 500 ppm
Sep 29 17:53:14 dist-test-slave-53nh krb5kdc[2800](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704394, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:53:14.800726 2907 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260929 17:53:14.801625 2907 webserver.cc:460] Webserver started at http://127.0.98.1:43589/ using document root <none> and password file <none>
I20260929 17:53:14.802132 2907 fs_manager.cc:359] Metadata directory not provided
I20260929 17:53:14.802196 2907 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:53:14.802371 2907 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:53:14.804330 2907 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704375975685-392-0/minicluster-data/ts-0/data/instance:
uuid: "4a68b84203384881a55cb93717dc1b9c"
format_stamp: "Formatted at 2026-09-29 17:53:14 on dist-test-slave-53nh"
server_key: "d563788de42008761038357855f7d5ec"
server_key_iv: "2645ecdacc9a44615129d23fc4600f20"
server_key_version: "encryptionkey@0"
I20260929 17:53:14.804840 2907 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704375975685-392-0/minicluster-data/ts-0/wal/instance:
uuid: "4a68b84203384881a55cb93717dc1b9c"
format_stamp: "Formatted at 2026-09-29 17:53:14 on dist-test-slave-53nh"
server_key: "d563788de42008761038357855f7d5ec"
server_key_iv: "2645ecdacc9a44615129d23fc4600f20"
server_key_version: "encryptionkey@0"
I20260929 17:53:14.808247 2907 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260929 17:53:14.810496 2922 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:53:14.811638 2907 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 17:53:14.811764 2907 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704375975685-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704375975685-392-0/minicluster-data/ts-0/wal
uuid: "4a68b84203384881a55cb93717dc1b9c"
format_stamp: "Formatted at 2026-09-29 17:53:14 on dist-test-slave-53nh"
server_key: "d563788de42008761038357855f7d5ec"
server_key_iv: "2645ecdacc9a44615129d23fc4600f20"
server_key_version: "encryptionkey@0"
I20260929 17:53:14.811851 2907 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704375975685-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704375975685-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704375975685-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:53:14.842402 2907 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:53:14.845424 2907 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:53:14.845614 2907 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:53:14.846194 2907 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 17:53:14.846998 2907 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:53:14.847054 2907 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:53:14.847105 2907 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:53:14.847129 2907 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 17:53:14.855875 2907 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33073
I20260929 17:53:14.855907 3035 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33073 every 8 connection(s)
I20260929 17:53:14.856742 2907 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704375975685-392-0/minicluster-data/ts-0/data/info.pb
I20260929 17:53:14.864019 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskup3Soe/build/debug/bin/kudu as pid 2907
I20260929 17:53:14.864127 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704375975685-392-0/minicluster-data/ts-0/wal/instance
I20260929 17:53:14.864436 392 external_mini_cluster.cc:1428] Setting key ff4952a7ce0a225c3a121f527fddffc6
Sep 29 17:53:14 dist-test-slave-53nh krb5kdc[2800](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704394, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 17:53:14.870114 2884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:14.858414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42785 (local address 127.0.98.62:38059)
0929 17:53:14.858569 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:14.858574 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:14.859454 (+ 880us) server_negotiation.cc:407] Connection header received
0929 17:53:14.860751 (+ 1297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:14.860754 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:14.860819 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:14.860892 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:53:14.862027 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:14.862556 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:14.863620 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:14.863865 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:14.863947 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:14.866686 (+ 2739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:53:14.866689 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:53:14.866691 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:53:14.866715 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:53:14.868105 (+ 1390us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:53:14.868596 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:53:14.868599 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:53:14.868601 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:53:14.868671 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:53:14.868973 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:53:14.868976 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:53:14.868978 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:53:14.869187 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:53:14.869271 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0929 17:53:14.869828 (+ 557us) server_negotiation.cc:299] Negotiation successful
0929 17:53:14.869954 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 17:53:14.870908 3038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:14.858843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38059 (local address 127.0.98.1:42785)
0929 17:53:14.859290 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:14.859351 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0929 17:53:14.860562 (+ 1211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:53:14.861070 (+ 508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:53:14.861077 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:53:14.861438 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:53:14.861900 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:14.861914 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:14.862645 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:14.862647 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:14.863517 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:14.863526 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:14.864019 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:14.864023 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:14.864215 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:14.864798 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:53:14.864835 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:53:14.866538 (+ 1703us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:53:14.868215 (+ 1677us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:53:14.868221 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:53:14.868236 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:53:14.868505 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:53:14.868749 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:53:14.868753 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:53:14.868755 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:53:14.868890 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:53:14.869282 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:53:14.869289 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:53:14.869596 (+ 307us) client_negotiation.cc:769] Sending connection context
0929 17:53:14.869768 (+ 172us) client_negotiation.cc:240] Negotiation successful
0929 17:53:14.870091 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":116,"threads_started":1}
I20260929 17:53:14.872067 3036 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38059
I20260929 17:53:14.872340 3036 heartbeater.cc:464] Registering TS with master...
I20260929 17:53:14.872936 3036 heartbeater.cc:511] Master 127.0.98.62:38059 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260929 17:53:14.874557 2845 ts_manager.cc:194] Registered new tserver with Master: 4a68b84203384881a55cb93717dc1b9c (127.0.98.1:33073)
I20260929 17:53:14.875672 2845 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:42785
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704375975685-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704375975685-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260929 17:53:14.889989 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskup3Soe/build/debug/bin/kudu
/tmp/dist-test-taskup3Soe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704375975685-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704375975685-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704375975685-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704375975685-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38059
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704375975685-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704375975685-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704375975685-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704375975685-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:53:14.995816 3041 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:53:14.996070 3041 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:53:14.996114 3041 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:53:15.000074 3041 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:53:15.000140 3041 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:53:15.000197 3041 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 17:53:15.005501 3041 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704375975685-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704375975685-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704375975685-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704375975685-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38059
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3041
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704375975685-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:53:15.006542 3041 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:53:15.007548 3041 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:53:15.017660 3046 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:53:15.017671 3047 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:53:15.017674 3049 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:53:15.017946 3041 server_base.cc:1034] running on GCE node
I20260929 17:53:15.018349 3041 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:53:15.018997 3041 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:53:15.020172 3041 hybrid_clock.cc:648] HybridClock initialized: now 1790704395020148 us; error 37 us; skew 500 ppm
Sep 29 17:53:15 dist-test-slave-53nh krb5kdc[2800](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704395, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:53:15.023638 3041 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260929 17:53:15.024632 3041 webserver.cc:460] Webserver started at http://127.0.98.2:45617/ using document root <none> and password file <none>
I20260929 17:53:15.025233 3041 fs_manager.cc:359] Metadata directory not provided
I20260929 17:53:15.025310 3041 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:53:15.025539 3041 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:53:15.028026 3041 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704375975685-392-0/minicluster-data/ts-1/data/instance:
uuid: "46a7d053caaf4e4d8ed9a86b895b0375"
format_stamp: "Formatted at 2026-09-29 17:53:15 on dist-test-slave-53nh"
server_key: "4c73c694c27fbc61e24c3758e4f906f1"
server_key_iv: "f83251382c175acbb6afa4d0659c1d9f"
server_key_version: "encryptionkey@0"
I20260929 17:53:15.028604 3041 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704375975685-392-0/minicluster-data/ts-1/wal/instance:
uuid: "46a7d053caaf4e4d8ed9a86b895b0375"
format_stamp: "Formatted at 2026-09-29 17:53:15 on dist-test-slave-53nh"
server_key: "4c73c694c27fbc61e24c3758e4f906f1"
server_key_iv: "f83251382c175acbb6afa4d0659c1d9f"
server_key_version: "encryptionkey@0"
I20260929 17:53:15.032133 3041 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260929 17:53:15.034528 3056 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:53:15.035696 3041 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 17:53:15.035801 3041 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704375975685-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704375975685-392-0/minicluster-data/ts-1/wal
uuid: "46a7d053caaf4e4d8ed9a86b895b0375"
format_stamp: "Formatted at 2026-09-29 17:53:15 on dist-test-slave-53nh"
server_key: "4c73c694c27fbc61e24c3758e4f906f1"
server_key_iv: "f83251382c175acbb6afa4d0659c1d9f"
server_key_version: "encryptionkey@0"
I20260929 17:53:15.035889 3041 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704375975685-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704375975685-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704375975685-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:53:15.052505 3041 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:53:15.055274 3041 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:53:15.055456 3041 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:53:15.056056 3041 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 17:53:15.056855 3041 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:53:15.056902 3041 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:53:15.056955 3041 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:53:15.056979 3041 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 17:53:15.065090 3041 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38229
I20260929 17:53:15.065120 3169 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38229 every 8 connection(s)
I20260929 17:53:15.065951 3041 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704375975685-392-0/minicluster-data/ts-1/data/info.pb
Sep 29 17:53:15 dist-test-slave-53nh krb5kdc[2800](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704395, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 17:53:15.075613 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskup3Soe/build/debug/bin/kudu as pid 3041
I20260929 17:53:15.075722 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704375975685-392-0/minicluster-data/ts-1/wal/instance
I20260929 17:53:15.076020 392 external_mini_cluster.cc:1428] Setting key 6659ecbee855964bc8661d72ced32cdb
I20260929 17:53:15.079751 2884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:15.067744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35671 (local address 127.0.98.62:38059)
0929 17:53:15.067865 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:15.067869 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:15.068723 (+ 854us) server_negotiation.cc:407] Connection header received
0929 17:53:15.070077 (+ 1354us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:15.070080 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:15.070132 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:15.070206 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:53:15.071386 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:15.071928 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:15.073121 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:15.073440 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:15.073532 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:15.076019 (+ 2487us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:53:15.076022 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:53:15.076024 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:53:15.076054 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:53:15.077569 (+ 1515us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:53:15.078147 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:53:15.078149 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:53:15.078151 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:53:15.078222 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:53:15.078565 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:53:15.078568 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:53:15.078569 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:53:15.078781 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:53:15.078856 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0929 17:53:15.079460 (+ 604us) server_negotiation.cc:299] Negotiation successful
0929 17:53:15.079589 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260929 17:53:15.080524 3172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:15.068167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38059 (local address 127.0.98.2:35671)
0929 17:53:15.068589 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:15.068630 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 17:53:15.069885 (+ 1255us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:53:15.070380 (+ 495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:53:15.070387 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:53:15.070771 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:53:15.071245 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:15.071258 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:15.072046 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:15.072049 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:15.072981 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:15.072990 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:15.073572 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:15.073575 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:15.073717 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:15.074274 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:53:15.074290 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:53:15.075805 (+ 1515us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:53:15.077676 (+ 1871us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:53:15.077683 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:53:15.077699 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:53:15.078066 (+ 367us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:53:15.078293 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:53:15.078296 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:53:15.078298 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:53:15.078474 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:53:15.078870 (+ 396us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:53:15.078876 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:53:15.079217 (+ 341us) client_negotiation.cc:769] Sending connection context
0929 17:53:15.079384 (+ 167us) client_negotiation.cc:240] Negotiation successful
0929 17:53:15.079648 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":106,"threads_started":1}
I20260929 17:53:15.081712 3170 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38059
I20260929 17:53:15.081949 3170 heartbeater.cc:464] Registering TS with master...
I20260929 17:53:15.082410 3170 heartbeater.cc:511] Master 127.0.98.62:38059 requested a full tablet report, sending...
I20260929 17:53:15.083473 2845 ts_manager.cc:194] Registered new tserver with Master: 46a7d053caaf4e4d8ed9a86b895b0375 (127.0.98.2:38229)
I20260929 17:53:15.084261 2845 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:35671
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704375975685-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704375975685-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260929 17:53:15.104651 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskup3Soe/build/debug/bin/kudu
/tmp/dist-test-taskup3Soe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704375975685-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704375975685-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704375975685-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704375975685-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38059
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704375975685-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704375975685-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704375975685-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704375975685-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:53:15.216643 3175 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:53:15.216907 3175 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:53:15.216949 3175 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:53:15.220963 3175 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:53:15.221052 3175 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:53:15.221119 3175 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 17:53:15.226409 3175 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704375975685-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704375975685-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704375975685-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704375975685-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38059
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3175
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704375975685-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:53:15.227414 3175 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:53:15.228353 3175 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:53:15.237452 3181 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:53:15.237464 3180 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:53:15.237573 3175 server_base.cc:1034] running on GCE node
W20260929 17:53:15.237468 3183 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:53:15.238195 3175 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:53:15.238862 3175 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:53:15.240028 3175 hybrid_clock.cc:648] HybridClock initialized: now 1790704395240003 us; error 33 us; skew 500 ppm
Sep 29 17:53:15 dist-test-slave-53nh krb5kdc[2800](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704395, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:53:15.242810 3175 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260929 17:53:15.243716 3175 webserver.cc:460] Webserver started at http://127.0.98.3:38723/ using document root <none> and password file <none>
I20260929 17:53:15.244258 3175 fs_manager.cc:359] Metadata directory not provided
I20260929 17:53:15.244339 3175 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:53:15.244508 3175 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:53:15.246531 3175 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704375975685-392-0/minicluster-data/ts-2/data/instance:
uuid: "68b009b6a61742c0acf0770b379ef4a9"
format_stamp: "Formatted at 2026-09-29 17:53:15 on dist-test-slave-53nh"
server_key: "a98df4ae98d6bfa8c06c7cf53a2babee"
server_key_iv: "7af97358c9fd283fcaea69340e5acc97"
server_key_version: "encryptionkey@0"
I20260929 17:53:15.247033 3175 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704375975685-392-0/minicluster-data/ts-2/wal/instance:
uuid: "68b009b6a61742c0acf0770b379ef4a9"
format_stamp: "Formatted at 2026-09-29 17:53:15 on dist-test-slave-53nh"
server_key: "a98df4ae98d6bfa8c06c7cf53a2babee"
server_key_iv: "7af97358c9fd283fcaea69340e5acc97"
server_key_version: "encryptionkey@0"
I20260929 17:53:15.250895 3175 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 17:53:15.253398 3190 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:53:15.254650 3175 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260929 17:53:15.254760 3175 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704375975685-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704375975685-392-0/minicluster-data/ts-2/wal
uuid: "68b009b6a61742c0acf0770b379ef4a9"
format_stamp: "Formatted at 2026-09-29 17:53:15 on dist-test-slave-53nh"
server_key: "a98df4ae98d6bfa8c06c7cf53a2babee"
server_key_iv: "7af97358c9fd283fcaea69340e5acc97"
server_key_version: "encryptionkey@0"
I20260929 17:53:15.254863 3175 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704375975685-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704375975685-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704375975685-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:53:15.277105 3175 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:53:15.280285 3175 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:53:15.280508 3175 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:53:15.281142 3175 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 17:53:15.281958 3175 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:53:15.282003 3175 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:53:15.282040 3175 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:53:15.282059 3175 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 17:53:15.290108 3175 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40957
I20260929 17:53:15.290134 3303 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40957 every 8 connection(s)
I20260929 17:53:15.291009 3175 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704375975685-392-0/minicluster-data/ts-2/data/info.pb
Sep 29 17:53:15 dist-test-slave-53nh krb5kdc[2800](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704395, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 17:53:15.301077 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskup3Soe/build/debug/bin/kudu as pid 3175
I20260929 17:53:15.301174 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790704375975685-392-0/minicluster-data/ts-2/wal/instance
I20260929 17:53:15.301425 392 external_mini_cluster.cc:1428] Setting key 83a7de84b2fc9582ea4656df100181c4
I20260929 17:53:15.304189 2884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:15.292720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36513 (local address 127.0.98.62:38059)
0929 17:53:15.292853 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:15.292857 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:15.293727 (+ 870us) server_negotiation.cc:407] Connection header received
0929 17:53:15.295099 (+ 1372us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:15.295103 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:15.295158 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:15.295231 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:53:15.296417 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:15.296936 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:15.298057 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:15.298274 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:15.298359 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:15.300686 (+ 2327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:53:15.300689 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:53:15.300691 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:53:15.300717 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:53:15.302188 (+ 1471us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:53:15.302778 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:53:15.302782 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:53:15.302783 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:53:15.302871 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:53:15.303162 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:53:15.303165 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:53:15.303166 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:53:15.303414 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:53:15.303508 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0929 17:53:15.303983 (+ 475us) server_negotiation.cc:299] Negotiation successful
0929 17:53:15.304077 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 17:53:15.304991 3306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:15.293146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38059 (local address 127.0.98.3:36513)
0929 17:53:15.293576 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:15.293618 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 17:53:15.294886 (+ 1268us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:53:15.295449 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:53:15.295457 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:53:15.295808 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:53:15.296277 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:15.296290 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:15.297065 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:15.297068 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:15.297933 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:15.297942 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:15.298407 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:15.298411 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:15.298563 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:15.299099 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:53:15.299115 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:53:15.300525 (+ 1410us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:53:15.302337 (+ 1812us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:53:15.302343 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:53:15.302359 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:53:15.302684 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:53:15.302950 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:53:15.302952 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:53:15.302953 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:53:15.303086 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:53:15.303516 (+ 430us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:53:15.303522 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:53:15.303767 (+ 245us) client_negotiation.cc:769] Sending connection context
0929 17:53:15.303949 (+ 182us) client_negotiation.cc:240] Negotiation successful
0929 17:53:15.304212 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":107,"threads_started":1}
I20260929 17:53:15.305977 3304 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38059
I20260929 17:53:15.306191 3304 heartbeater.cc:464] Registering TS with master...
I20260929 17:53:15.306648 3304 heartbeater.cc:511] Master 127.0.98.62:38059 requested a full tablet report, sending...
I20260929 17:53:15.307536 2845 ts_manager.cc:194] Registered new tserver with Master: 68b009b6a61742c0acf0770b379ef4a9 (127.0.98.3:40957)
I20260929 17:53:15.308055 2845 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:36513
I20260929 17:53:15.315044 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260929 17:53:15.326536 2884 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:53:15.325104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54126 (local address 127.0.98.62:38059)
0929 17:53:15.325279 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:15.325284 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:15.325389 (+ 105us) server_negotiation.cc:407] Connection header received
0929 17:53:15.325506 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:15.325508 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:15.325549 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:15.325605 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:53:15.326393 (+ 788us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54126: BlockingRecv error: recv got EOF from 127.0.0.1:54126 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 17:53:15.327556 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskup3Soe/build/debug/bin/kudu with pid 2907
I20260929 17:53:15.332468 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskup3Soe/build/debug/bin/kudu with pid 3041
I20260929 17:53:15.336664 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskup3Soe/build/debug/bin/kudu with pid 3175
I20260929 17:53:15.341089 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskup3Soe/build/debug/bin/kudu with pid 2813
2026-09-29T17:53:15Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (1986 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790704375975685-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 17:53:15 dist-test-slave-53nh krb5kdc[3316](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 17:53:15 dist-test-slave-53nh krb5kdc[3316](info): set up 2 sockets
Sep 29 17:53:15 dist-test-slave-53nh krb5kdc[3316](info): commencing operation
krb5kdc: starting...
W20260929 17:53:16.288664 392 mini_kdc.cc:121] Time spent starting KDC: real 0.928s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790704375975685-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790704375975685-392-0/krb5kdc/test-user.keytab.
Sep 29 17:53:16 dist-test-slave-53nh krb5kdc[3316](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704396, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T17:53:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T17:53:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790704375975685-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790704375975685-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 17:53:16.399806 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskup3Soe/build/debug/bin/kudu
/tmp/dist-test-taskup3Soe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790704375975685-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790704375975685-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790704375975685-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790704375975685-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38605
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790704375975685-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37195
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38605
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790704375975685-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790704375975685-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790704375975685-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:53:16.511312 3329 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:53:16.511600 3329 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:53:16.511659 3329 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:53:16.515625 3329 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 17:53:16.515682 3329 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:53:16.515709 3329 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:53:16.515730 3329 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 17:53:16.515751 3329 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 17:53:16.520888 3329 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37195
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790704375975685-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790704375975685-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38605
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790704375975685-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38605
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790704375975685-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3329
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790704375975685-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:53:16.521912 3329 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:53:16.522917 3329 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:53:16.530956 3337 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:53:16.530961 3334 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:53:16.530998 3329 server_base.cc:1034] running on GCE node
W20260929 17:53:16.530961 3335 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:53:16.531600 3329 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:53:16.532640 3329 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:53:16.533820 3329 hybrid_clock.cc:648] HybridClock initialized: now 1790704396533811 us; error 30 us; skew 500 ppm
Sep 29 17:53:16 dist-test-slave-53nh krb5kdc[3316](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704396, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:53:16.539448 3329 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 17:53:16.540355 3329 webserver.cc:460] Webserver started at http://127.0.98.62:44083/ using document root <none> and password file <none>
I20260929 17:53:16.540925 3329 fs_manager.cc:359] Metadata directory not provided
I20260929 17:53:16.540993 3329 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:53:16.541206 3329 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:53:16.543222 3329 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790704375975685-392-0/minicluster-data/master-0/data/instance:
uuid: "72b6d23e0b3c4e21a80bfa9ca33ca4ac"
format_stamp: "Formatted at 2026-09-29 17:53:16 on dist-test-slave-53nh"
server_key: "c4406674ac98bcc751b75480113eac2b"
server_key_iv: "8dacf7225d3b60ac6a145db0c279be4c"
server_key_version: "encryptionkey@0"
I20260929 17:53:16.543776 3329 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790704375975685-392-0/minicluster-data/master-0/wal/instance:
uuid: "72b6d23e0b3c4e21a80bfa9ca33ca4ac"
format_stamp: "Formatted at 2026-09-29 17:53:16 on dist-test-slave-53nh"
server_key: "c4406674ac98bcc751b75480113eac2b"
server_key_iv: "8dacf7225d3b60ac6a145db0c279be4c"
server_key_version: "encryptionkey@0"
I20260929 17:53:16.547644 3329 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260929 17:53:16.550215 3344 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:53:16.551645 3329 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 17:53:16.551781 3329 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790704375975685-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790704375975685-392-0/minicluster-data/master-0/wal
uuid: "72b6d23e0b3c4e21a80bfa9ca33ca4ac"
format_stamp: "Formatted at 2026-09-29 17:53:16 on dist-test-slave-53nh"
server_key: "c4406674ac98bcc751b75480113eac2b"
server_key_iv: "8dacf7225d3b60ac6a145db0c279be4c"
server_key_version: "encryptionkey@0"
I20260929 17:53:16.551889 3329 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790704375975685-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790704375975685-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790704375975685-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:53:16.570060 3329 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:53:16.573231 3329 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:53:16.573406 3329 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:53:16.579931 3329 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38605
I20260929 17:53:16.580173 3396 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38605 every 8 connection(s)
I20260929 17:53:16.580907 3329 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790704375975685-392-0/minicluster-data/master-0/data/info.pb
I20260929 17:53:16.583644 3397 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:53:16.585404 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskup3Soe/build/debug/bin/kudu as pid 3329
I20260929 17:53:16.585537 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790704375975685-392-0/minicluster-data/master-0/wal/instance
I20260929 17:53:16.585839 392 external_mini_cluster.cc:1428] Setting key ee6a4c5e86b296ed7b9d7eaa3b148601
I20260929 17:53:16.590031 3397 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 72b6d23e0b3c4e21a80bfa9ca33ca4ac: Bootstrap starting.
I20260929 17:53:16.592511 3397 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 72b6d23e0b3c4e21a80bfa9ca33ca4ac: Neither blocks nor log segments found. Creating new log.
I20260929 17:53:16.593400 3397 log.cc:824] T 00000000000000000000000000000000 P 72b6d23e0b3c4e21a80bfa9ca33ca4ac: Log is configured to *not* fsync() on all Append() calls
Sep 29 17:53:16 dist-test-slave-53nh krb5kdc[3316](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704396, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 17:53:16.595469 3397 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 72b6d23e0b3c4e21a80bfa9ca33ca4ac: No bootstrap required, opened a new log
I20260929 17:53:16.600183 3400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:16.587084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36170 (local address 127.0.98.62:38605)
0929 17:53:16.587485 (+ 401us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:16.587500 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:16.587534 (+ 34us) server_negotiation.cc:407] Connection header received
0929 17:53:16.588142 (+ 608us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:16.588158 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:16.588477 (+ 319us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:16.588746 (+ 269us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:53:16.589737 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:16.590732 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:16.591946 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:16.592391 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:16.592497 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:16.595206 (+ 2709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:53:16.595213 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:53:16.595229 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:53:16.595261 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:53:16.597144 (+ 1883us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:53:16.597541 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:53:16.597547 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:53:16.597553 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:53:16.597716 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:53:16.597966 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:53:16.597970 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:53:16.597972 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:53:16.598485 (+ 513us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:53:16.598701 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0929 17:53:16.598886 (+ 185us) server_negotiation.cc:299] Negotiation successful
0929 17:53:16.599204 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":269,"thread_start_us":102,"threads_started":1}
I20260929 17:53:16.600019 3397 raft_consensus.cc:348] T 00000000000000000000000000000000 P 72b6d23e0b3c4e21a80bfa9ca33ca4ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72b6d23e0b3c4e21a80bfa9ca33ca4ac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38605 } }
I20260929 17:53:16.600275 3397 raft_consensus.cc:374] T 00000000000000000000000000000000 P 72b6d23e0b3c4e21a80bfa9ca33ca4ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:53:16.600322 3397 raft_consensus.cc:729] T 00000000000000000000000000000000 P 72b6d23e0b3c4e21a80bfa9ca33ca4ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72b6d23e0b3c4e21a80bfa9ca33ca4ac, State: Initialized, Role: FOLLOWER
I20260929 17:53:16.600785 3397 consensus_queue.cc:260] T 00000000000000000000000000000000 P 72b6d23e0b3c4e21a80bfa9ca33ca4ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72b6d23e0b3c4e21a80bfa9ca33ca4ac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38605 } }
I20260929 17:53:16.600921 3397 raft_consensus.cc:388] T 00000000000000000000000000000000 P 72b6d23e0b3c4e21a80bfa9ca33ca4ac [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 17:53:16.600991 3397 raft_consensus.cc:482] T 00000000000000000000000000000000 P 72b6d23e0b3c4e21a80bfa9ca33ca4ac [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 17:53:16.601112 3397 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 72b6d23e0b3c4e21a80bfa9ca33ca4ac [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:53:16.602005 3397 raft_consensus.cc:504] T 00000000000000000000000000000000 P 72b6d23e0b3c4e21a80bfa9ca33ca4ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72b6d23e0b3c4e21a80bfa9ca33ca4ac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38605 } }
I20260929 17:53:16.602355 3397 leader_election.cc:302] T 00000000000000000000000000000000 P 72b6d23e0b3c4e21a80bfa9ca33ca4ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 72b6d23e0b3c4e21a80bfa9ca33ca4ac; no voters:
I20260929 17:53:16.602648 3397 leader_election.cc:288] T 00000000000000000000000000000000 P 72b6d23e0b3c4e21a80bfa9ca33ca4ac [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 17:53:16.602741 3402 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 72b6d23e0b3c4e21a80bfa9ca33ca4ac [term 1 FOLLOWER]: Leader election won for term 1
I20260929 17:53:16.602981 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 17:53:16.602968 3402 raft_consensus.cc:686] T 00000000000000000000000000000000 P 72b6d23e0b3c4e21a80bfa9ca33ca4ac [term 1 LEADER]: Becoming Leader. State: Replica: 72b6d23e0b3c4e21a80bfa9ca33ca4ac, State: Running, Role: LEADER
I20260929 17:53:16.603253 3402 consensus_queue.cc:237] T 00000000000000000000000000000000 P 72b6d23e0b3c4e21a80bfa9ca33ca4ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72b6d23e0b3c4e21a80bfa9ca33ca4ac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38605 } }
I20260929 17:53:16.603786 3397 sys_catalog.cc:564] T 00000000000000000000000000000000 P 72b6d23e0b3c4e21a80bfa9ca33ca4ac [sys.catalog]: configured and running, proceeding with master startup.
I20260929 17:53:16.604434 3402 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72b6d23e0b3c4e21a80bfa9ca33ca4ac [sys.catalog]: SysCatalogTable state changed. Reason: New leader 72b6d23e0b3c4e21a80bfa9ca33ca4ac. Latest consensus state: current_term: 1 leader_uuid: "72b6d23e0b3c4e21a80bfa9ca33ca4ac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72b6d23e0b3c4e21a80bfa9ca33ca4ac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38605 } } }
I20260929 17:53:16.604532 3402 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72b6d23e0b3c4e21a80bfa9ca33ca4ac [sys.catalog]: This master's current role is: LEADER
I20260929 17:53:16.604833 3403 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72b6d23e0b3c4e21a80bfa9ca33ca4ac [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "72b6d23e0b3c4e21a80bfa9ca33ca4ac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72b6d23e0b3c4e21a80bfa9ca33ca4ac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38605 } } }
I20260929 17:53:16.604939 3403 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72b6d23e0b3c4e21a80bfa9ca33ca4ac [sys.catalog]: This master's current role is: LEADER
I20260929 17:53:16.605347 3409 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 17:53:16.608566 3409 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 17:53:16.614064 3409 catalog_manager.cc:1347] Generated new cluster ID: 492ec05cc56045d184577bf2e083ca16
I20260929 17:53:16.614190 3409 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 17:53:16.624097 3400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:16.618323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36178 (local address 127.0.98.62:38605)
0929 17:53:16.618463 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:16.618467 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:16.618849 (+ 382us) server_negotiation.cc:407] Connection header received
0929 17:53:16.618974 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:16.618978 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:16.619035 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:16.619137 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:53:16.619750 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:16.620492 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:16.621633 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:16.622013 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:16.622131 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:16.623003 (+ 872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:53:16.623007 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:53:16.623011 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 17:53:16.623615 (+ 604us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:53:16.623655 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 17:53:16.623678 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:53:16.623779 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0929 17:53:16.623867 (+ 88us) server_negotiation.cc:299] Negotiation successful
0929 17:53:16.623933 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 17:53:16.630040 3409 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 17:53:16.631275 3409 catalog_manager.cc:1504] Loading token signing keys...
I20260929 17:53:16.654135 3409 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 72b6d23e0b3c4e21a80bfa9ca33ca4ac: Generated new TSK 0
I20260929 17:53:16.654979 3409 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 17:53:16.669108 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskup3Soe/build/debug/bin/kudu with pid 3329
2026-09-29T17:53:16Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1326 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704375975685-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 17:53:16 dist-test-slave-53nh krb5kdc[3430](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 17:53:16 dist-test-slave-53nh krb5kdc[3430](info): set up 2 sockets
Sep 29 17:53:16 dist-test-slave-53nh krb5kdc[3430](info): commencing operation
krb5kdc: starting...
W20260929 17:53:17.612715 392 mini_kdc.cc:121] Time spent starting KDC: real 0.925s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704375975685-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704375975685-392-0/krb5kdc/test-user.keytab.
Sep 29 17:53:17 dist-test-slave-53nh krb5kdc[3430](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704397, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T17:53:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T17:53:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704375975685-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704375975685-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 17:53:17.725090 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskup3Soe/build/debug/bin/kudu
/tmp/dist-test-taskup3Soe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704375975685-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704375975685-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704375975685-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704375975685-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42299
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704375975685-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42299
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704375975685-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704375975685-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704375975685-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:53:17.830884 3443 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:53:17.831142 3443 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:53:17.831179 3443 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:53:17.835125 3443 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 17:53:17.835184 3443 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:53:17.835202 3443 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:53:17.835217 3443 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 17:53:17.835250 3443 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 17:53:17.840463 3443 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704375975685-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704375975685-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42299
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704375975685-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42299
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704375975685-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3443
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704375975685-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:53:17.841496 3443 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:53:17.842476 3443 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:53:17.850332 3449 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:53:17.850343 3448 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:53:17.850390 3451 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:53:17.850473 3443 server_base.cc:1034] running on GCE node
I20260929 17:53:17.851126 3443 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:53:17.852141 3443 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:53:17.853287 3443 hybrid_clock.cc:648] HybridClock initialized: now 1790704397853274 us; error 29 us; skew 500 ppm
Sep 29 17:53:17 dist-test-slave-53nh krb5kdc[3430](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704397, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:53:17.855849 3443 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 17:53:17.856724 3443 webserver.cc:460] Webserver started at http://127.0.98.62:35497/ using document root <none> and password file <none>
I20260929 17:53:17.857282 3443 fs_manager.cc:359] Metadata directory not provided
I20260929 17:53:17.857344 3443 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:53:17.857512 3443 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:53:17.859426 3443 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704375975685-392-0/minicluster-data/master-0/data/instance:
uuid: "9ffe48c676a94e8b86c4c492d0bd9ff4"
format_stamp: "Formatted at 2026-09-29 17:53:17 on dist-test-slave-53nh"
server_key: "222329fc2f7e53681d9d2005c72daf8d"
server_key_iv: "f2f7168ab50486190ade769f1650b156"
server_key_version: "encryptionkey@0"
I20260929 17:53:17.859898 3443 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704375975685-392-0/minicluster-data/master-0/wal/instance:
uuid: "9ffe48c676a94e8b86c4c492d0bd9ff4"
format_stamp: "Formatted at 2026-09-29 17:53:17 on dist-test-slave-53nh"
server_key: "222329fc2f7e53681d9d2005c72daf8d"
server_key_iv: "f2f7168ab50486190ade769f1650b156"
server_key_version: "encryptionkey@0"
I20260929 17:53:17.863376 3443 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 17:53:17.865578 3458 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:53:17.866732 3443 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 17:53:17.866822 3443 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704375975685-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704375975685-392-0/minicluster-data/master-0/wal
uuid: "9ffe48c676a94e8b86c4c492d0bd9ff4"
format_stamp: "Formatted at 2026-09-29 17:53:17 on dist-test-slave-53nh"
server_key: "222329fc2f7e53681d9d2005c72daf8d"
server_key_iv: "f2f7168ab50486190ade769f1650b156"
server_key_version: "encryptionkey@0"
I20260929 17:53:17.866901 3443 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704375975685-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704375975685-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704375975685-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:53:17.886422 3443 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:53:17.889523 3443 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:53:17.889670 3443 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:53:17.896136 3443 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42299
I20260929 17:53:17.896427 3510 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42299 every 8 connection(s)
I20260929 17:53:17.897065 3443 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704375975685-392-0/minicluster-data/master-0/data/info.pb
I20260929 17:53:17.899749 3511 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:53:17.900863 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskup3Soe/build/debug/bin/kudu as pid 3443
I20260929 17:53:17.900949 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704375975685-392-0/minicluster-data/master-0/wal/instance
I20260929 17:53:17.901208 392 external_mini_cluster.cc:1428] Setting key 080903d60554794237b70a2fed0785a7
I20260929 17:53:17.905483 3511 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ffe48c676a94e8b86c4c492d0bd9ff4: Bootstrap starting.
I20260929 17:53:17.907883 3511 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9ffe48c676a94e8b86c4c492d0bd9ff4: Neither blocks nor log segments found. Creating new log.
I20260929 17:53:17.908708 3511 log.cc:824] T 00000000000000000000000000000000 P 9ffe48c676a94e8b86c4c492d0bd9ff4: Log is configured to *not* fsync() on all Append() calls
Sep 29 17:53:17 dist-test-slave-53nh krb5kdc[3430](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704397, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 17:53:17.910620 3511 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ffe48c676a94e8b86c4c492d0bd9ff4: No bootstrap required, opened a new log
I20260929 17:53:17.914927 3514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:17.902480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32790 (local address 127.0.98.62:42299)
0929 17:53:17.902835 (+ 355us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:17.902849 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:17.902882 (+ 33us) server_negotiation.cc:407] Connection header received
0929 17:53:17.903480 (+ 598us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:17.903497 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:17.903811 (+ 314us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:17.904084 (+ 273us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:53:17.904934 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:17.905951 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:17.907039 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:17.907481 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:17.907593 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:17.910120 (+ 2527us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:53:17.910126 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:53:17.910141 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:53:17.910169 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:53:17.911999 (+ 1830us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:53:17.912377 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:53:17.912383 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:53:17.912390 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:53:17.912524 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:53:17.912754 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:53:17.912757 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:53:17.912758 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:53:17.913285 (+ 527us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:53:17.913462 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0929 17:53:17.913731 (+ 269us) server_negotiation.cc:299] Negotiation successful
0929 17:53:17.914007 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":225,"thread_start_us":88,"threads_started":1}
I20260929 17:53:17.914867 3511 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9ffe48c676a94e8b86c4c492d0bd9ff4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ffe48c676a94e8b86c4c492d0bd9ff4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42299 } }
I20260929 17:53:17.915115 3511 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9ffe48c676a94e8b86c4c492d0bd9ff4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:53:17.915175 3511 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9ffe48c676a94e8b86c4c492d0bd9ff4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ffe48c676a94e8b86c4c492d0bd9ff4, State: Initialized, Role: FOLLOWER
I20260929 17:53:17.915618 3511 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9ffe48c676a94e8b86c4c492d0bd9ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ffe48c676a94e8b86c4c492d0bd9ff4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42299 } }
I20260929 17:53:17.915767 3511 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9ffe48c676a94e8b86c4c492d0bd9ff4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 17:53:17.915827 3511 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9ffe48c676a94e8b86c4c492d0bd9ff4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 17:53:17.915915 3511 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9ffe48c676a94e8b86c4c492d0bd9ff4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:53:17.916922 3511 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9ffe48c676a94e8b86c4c492d0bd9ff4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ffe48c676a94e8b86c4c492d0bd9ff4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42299 } }
I20260929 17:53:17.917259 3511 leader_election.cc:302] T 00000000000000000000000000000000 P 9ffe48c676a94e8b86c4c492d0bd9ff4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9ffe48c676a94e8b86c4c492d0bd9ff4; no voters:
I20260929 17:53:17.917517 3511 leader_election.cc:288] T 00000000000000000000000000000000 P 9ffe48c676a94e8b86c4c492d0bd9ff4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 17:53:17.917685 3516 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9ffe48c676a94e8b86c4c492d0bd9ff4 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 17:53:17.917876 3516 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9ffe48c676a94e8b86c4c492d0bd9ff4 [term 1 LEADER]: Becoming Leader. State: Replica: 9ffe48c676a94e8b86c4c492d0bd9ff4, State: Running, Role: LEADER
I20260929 17:53:17.918169 3516 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9ffe48c676a94e8b86c4c492d0bd9ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ffe48c676a94e8b86c4c492d0bd9ff4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42299 } }
I20260929 17:53:17.918599 3511 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9ffe48c676a94e8b86c4c492d0bd9ff4 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 17:53:17.919574 3517 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ffe48c676a94e8b86c4c492d0bd9ff4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9ffe48c676a94e8b86c4c492d0bd9ff4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ffe48c676a94e8b86c4c492d0bd9ff4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42299 } } }
I20260929 17:53:17.919695 3517 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ffe48c676a94e8b86c4c492d0bd9ff4 [sys.catalog]: This master's current role is: LEADER
I20260929 17:53:17.919962 3518 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ffe48c676a94e8b86c4c492d0bd9ff4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9ffe48c676a94e8b86c4c492d0bd9ff4. Latest consensus state: current_term: 1 leader_uuid: "9ffe48c676a94e8b86c4c492d0bd9ff4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ffe48c676a94e8b86c4c492d0bd9ff4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42299 } } }
I20260929 17:53:17.920043 3525 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 17:53:17.920050 3518 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ffe48c676a94e8b86c4c492d0bd9ff4 [sys.catalog]: This master's current role is: LEADER
I20260929 17:53:17.923003 3525 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 17:53:17.928285 3525 catalog_manager.cc:1347] Generated new cluster ID: a0fcc5aa556d4d8fbb78b72adf43be64
I20260929 17:53:17.928350 3525 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 17:53:17.975605 3525 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 17:53:17.977324 3525 catalog_manager.cc:1504] Loading token signing keys...
I20260929 17:53:17.988354 3525 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9ffe48c676a94e8b86c4c492d0bd9ff4: Generated new TSK 0
I20260929 17:53:17.989267 3525 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704375975685-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704375975685-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260929 17:53:18.020244 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskup3Soe/build/debug/bin/kudu
/tmp/dist-test-taskup3Soe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704375975685-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704375975685-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704375975685-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704375975685-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42299
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704375975685-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704375975685-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704375975685-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704375975685-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:53:18.126535 3537 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:53:18.126780 3537 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:53:18.126847 3537 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:53:18.130770 3537 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:53:18.130829 3537 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:53:18.130908 3537 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 17:53:18.135990 3537 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704375975685-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704375975685-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704375975685-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704375975685-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42299
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3537
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704375975685-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:53:18.136952 3537 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:53:18.137909 3537 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:53:18.147151 3545 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:53:18.147321 3542 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:53:18.147186 3537 server_base.cc:1034] running on GCE node
W20260929 17:53:18.147195 3543 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:53:18.147794 3537 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:53:18.148483 3537 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:53:18.149678 3537 hybrid_clock.cc:648] HybridClock initialized: now 1790704398149641 us; error 49 us; skew 500 ppm
Sep 29 17:53:18 dist-test-slave-53nh krb5kdc[3430](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704398, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:53:18.152407 3537 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260929 17:53:18.153326 3537 webserver.cc:460] Webserver started at http://127.0.98.1:38333/ using document root <none> and password file <none>
I20260929 17:53:18.153836 3537 fs_manager.cc:359] Metadata directory not provided
I20260929 17:53:18.153889 3537 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:53:18.154050 3537 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:53:18.155975 3537 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704375975685-392-0/minicluster-data/ts-0/data/instance:
uuid: "62385847aba74bb2be4fa9b8e4b20125"
format_stamp: "Formatted at 2026-09-29 17:53:18 on dist-test-slave-53nh"
server_key: "73d0fa5cd192d3f5e0fbbb856cc71838"
server_key_iv: "1f494d22e5c58e6644f9fef767e96eea"
server_key_version: "encryptionkey@0"
I20260929 17:53:18.156450 3537 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704375975685-392-0/minicluster-data/ts-0/wal/instance:
uuid: "62385847aba74bb2be4fa9b8e4b20125"
format_stamp: "Formatted at 2026-09-29 17:53:18 on dist-test-slave-53nh"
server_key: "73d0fa5cd192d3f5e0fbbb856cc71838"
server_key_iv: "1f494d22e5c58e6644f9fef767e96eea"
server_key_version: "encryptionkey@0"
I20260929 17:53:18.160008 3537 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260929 17:53:18.162436 3552 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:53:18.163589 3537 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 17:53:18.163682 3537 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704375975685-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704375975685-392-0/minicluster-data/ts-0/wal
uuid: "62385847aba74bb2be4fa9b8e4b20125"
format_stamp: "Formatted at 2026-09-29 17:53:18 on dist-test-slave-53nh"
server_key: "73d0fa5cd192d3f5e0fbbb856cc71838"
server_key_iv: "1f494d22e5c58e6644f9fef767e96eea"
server_key_version: "encryptionkey@0"
I20260929 17:53:18.163782 3537 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704375975685-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704375975685-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704375975685-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:53:18.183696 3537 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:53:18.186421 3537 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:53:18.186582 3537 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:53:18.187139 3537 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 17:53:18.187912 3537 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:53:18.187958 3537 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:53:18.187996 3537 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:53:18.188026 3537 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 17:53:18.196044 3537 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45017
I20260929 17:53:18.196108 3665 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45017 every 8 connection(s)
I20260929 17:53:18.196885 3537 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704375975685-392-0/minicluster-data/ts-0/data/info.pb
Sep 29 17:53:18 dist-test-slave-53nh krb5kdc[3430](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704398, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 17:53:18.206218 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskup3Soe/build/debug/bin/kudu as pid 3537
I20260929 17:53:18.206314 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704375975685-392-0/minicluster-data/ts-0/wal/instance
I20260929 17:53:18.206602 392 external_mini_cluster.cc:1428] Setting key 59fad076fbb8f9dfcad191af46ed3212
I20260929 17:53:18.209885 3514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:18.198560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35113 (local address 127.0.98.62:42299)
0929 17:53:18.198679 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:18.198683 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:18.199507 (+ 824us) server_negotiation.cc:407] Connection header received
0929 17:53:18.200817 (+ 1310us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:18.200820 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:18.200867 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:18.200942 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:53:18.202103 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:18.202643 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:18.203693 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:18.203927 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:18.204004 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:18.206243 (+ 2239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:53:18.206246 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:53:18.206248 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:53:18.206272 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:53:18.207780 (+ 1508us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:53:18.208368 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:53:18.208371 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:53:18.208373 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:53:18.208448 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:53:18.208784 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:53:18.208786 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:53:18.208787 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:53:18.208953 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:53:18.209054 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0929 17:53:18.209565 (+ 511us) server_negotiation.cc:299] Negotiation successful
0929 17:53:18.209702 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260929 17:53:18.210693 3668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:18.198897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42299 (local address 127.0.98.1:35113)
0929 17:53:18.199344 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:18.199385 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 17:53:18.200604 (+ 1219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:53:18.201126 (+ 522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:53:18.201133 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:53:18.201472 (+ 339us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:53:18.201951 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:18.201964 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:18.202733 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:18.202736 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:18.203589 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:18.203597 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:18.204042 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:18.204044 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:18.204175 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:18.204680 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:53:18.204694 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:53:18.206088 (+ 1394us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:53:18.207947 (+ 1859us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:53:18.207952 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:53:18.207962 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:53:18.208277 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:53:18.208529 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:53:18.208533 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:53:18.208534 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:53:18.208707 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:53:18.209049 (+ 342us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:53:18.209056 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:53:18.209341 (+ 285us) client_negotiation.cc:769] Sending connection context
0929 17:53:18.209532 (+ 191us) client_negotiation.cc:240] Negotiation successful
0929 17:53:18.209858 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":135,"threads_started":1}
I20260929 17:53:18.211803 3666 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42299
I20260929 17:53:18.212009 3666 heartbeater.cc:464] Registering TS with master...
I20260929 17:53:18.212443 3666 heartbeater.cc:511] Master 127.0.98.62:42299 requested a full tablet report, sending...
I20260929 17:53:18.213800 3475 ts_manager.cc:194] Registered new tserver with Master: 62385847aba74bb2be4fa9b8e4b20125 (127.0.98.1:45017)
I20260929 17:53:18.214851 3475 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:35113
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704375975685-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704375975685-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260929 17:53:18.229828 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskup3Soe/build/debug/bin/kudu
/tmp/dist-test-taskup3Soe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704375975685-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704375975685-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704375975685-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704375975685-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42299
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704375975685-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704375975685-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704375975685-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704375975685-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:53:18.334445 3671 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:53:18.334695 3671 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:53:18.334736 3671 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:53:18.338624 3671 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:53:18.338682 3671 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:53:18.338738 3671 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 17:53:18.343736 3671 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704375975685-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704375975685-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704375975685-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704375975685-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42299
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3671
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704375975685-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:53:18.344659 3671 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:53:18.345571 3671 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:53:18.354380 3677 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:53:18.354384 3676 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:53:18.354511 3671 server_base.cc:1034] running on GCE node
W20260929 17:53:18.354399 3679 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:53:18.354974 3671 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:53:18.355573 3671 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:53:18.356726 3671 hybrid_clock.cc:648] HybridClock initialized: now 1790704398356704 us; error 33 us; skew 500 ppm
Sep 29 17:53:18 dist-test-slave-53nh krb5kdc[3430](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704398, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:53:18.359324 3671 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260929 17:53:18.360184 3671 webserver.cc:460] Webserver started at http://127.0.98.2:39009/ using document root <none> and password file <none>
I20260929 17:53:18.360699 3671 fs_manager.cc:359] Metadata directory not provided
I20260929 17:53:18.360761 3671 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:53:18.360958 3671 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:53:18.362914 3671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704375975685-392-0/minicluster-data/ts-1/data/instance:
uuid: "50d02d2b584141ac89da5a15149f2027"
format_stamp: "Formatted at 2026-09-29 17:53:18 on dist-test-slave-53nh"
server_key: "24ee94e62e1335a76f4a907cbeec8a9b"
server_key_iv: "fe4bd28484ef83e4c5a47d3b741fa1fe"
server_key_version: "encryptionkey@0"
I20260929 17:53:18.363425 3671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704375975685-392-0/minicluster-data/ts-1/wal/instance:
uuid: "50d02d2b584141ac89da5a15149f2027"
format_stamp: "Formatted at 2026-09-29 17:53:18 on dist-test-slave-53nh"
server_key: "24ee94e62e1335a76f4a907cbeec8a9b"
server_key_iv: "fe4bd28484ef83e4c5a47d3b741fa1fe"
server_key_version: "encryptionkey@0"
I20260929 17:53:18.366953 3671 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 17:53:18.369262 3686 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:53:18.370432 3671 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260929 17:53:18.370539 3671 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704375975685-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704375975685-392-0/minicluster-data/ts-1/wal
uuid: "50d02d2b584141ac89da5a15149f2027"
format_stamp: "Formatted at 2026-09-29 17:53:18 on dist-test-slave-53nh"
server_key: "24ee94e62e1335a76f4a907cbeec8a9b"
server_key_iv: "fe4bd28484ef83e4c5a47d3b741fa1fe"
server_key_version: "encryptionkey@0"
I20260929 17:53:18.370625 3671 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704375975685-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704375975685-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704375975685-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:53:18.390209 3671 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:53:18.393138 3671 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:53:18.393306 3671 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:53:18.393857 3671 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 17:53:18.394644 3671 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:53:18.394688 3671 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:53:18.394737 3671 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:53:18.394764 3671 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 17:53:18.403020 3671 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45921
I20260929 17:53:18.403047 3799 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45921 every 8 connection(s)
I20260929 17:53:18.403887 3671 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704375975685-392-0/minicluster-data/ts-1/data/info.pb
I20260929 17:53:18.405222 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskup3Soe/build/debug/bin/kudu as pid 3671
I20260929 17:53:18.405355 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704375975685-392-0/minicluster-data/ts-1/wal/instance
I20260929 17:53:18.405642 392 external_mini_cluster.cc:1428] Setting key 0ec4becc04391f8d4560ba5694c6a0b1
Sep 29 17:53:18 dist-test-slave-53nh krb5kdc[3430](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704398, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260929 17:53:18.417100 3514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:18.405661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38197 (local address 127.0.98.62:42299)
0929 17:53:18.405818 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:18.405822 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:18.406712 (+ 890us) server_negotiation.cc:407] Connection header received
0929 17:53:18.408081 (+ 1369us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:18.408084 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:18.408137 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:18.408206 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:53:18.409347 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:18.409940 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:18.411041 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:18.411276 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:18.411362 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:18.413757 (+ 2395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:53:18.413759 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:53:18.413762 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:53:18.413785 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:53:18.415108 (+ 1323us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:53:18.415622 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:53:18.415625 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:53:18.415626 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:53:18.415695 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:53:18.416040 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:53:18.416044 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:53:18.416045 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:53:18.416221 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:53:18.416304 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0929 17:53:18.416795 (+ 491us) server_negotiation.cc:299] Negotiation successful
0929 17:53:18.416912 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260929 17:53:18.417973 3802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:18.406141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42299 (local address 127.0.98.2:38197)
0929 17:53:18.406573 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:18.406615 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 17:53:18.407883 (+ 1268us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:53:18.408367 (+ 484us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:53:18.408374 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:53:18.408708 (+ 334us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:53:18.409208 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:18.409221 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:18.410056 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:18.410059 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:18.410933 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:18.410942 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:18.411409 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:18.411413 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:18.411569 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:18.412155 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:53:18.412169 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:53:18.413557 (+ 1388us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:53:18.415213 (+ 1656us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:53:18.415254 (+ 41us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:53:18.415269 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:53:18.415508 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:53:18.415783 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:53:18.415786 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:53:18.415788 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:53:18.415957 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:53:18.416317 (+ 360us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:53:18.416324 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:53:18.416566 (+ 242us) client_negotiation.cc:769] Sending connection context
0929 17:53:18.416734 (+ 168us) client_negotiation.cc:240] Negotiation successful
0929 17:53:18.417094 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":116,"threads_started":1}
I20260929 17:53:18.419301 3800 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42299
I20260929 17:53:18.419584 3800 heartbeater.cc:464] Registering TS with master...
I20260929 17:53:18.420063 3800 heartbeater.cc:511] Master 127.0.98.62:42299 requested a full tablet report, sending...
I20260929 17:53:18.421177 3475 ts_manager.cc:194] Registered new tserver with Master: 50d02d2b584141ac89da5a15149f2027 (127.0.98.2:45921)
I20260929 17:53:18.421968 3475 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:38197
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704375975685-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704375975685-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260929 17:53:18.431643 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskup3Soe/build/debug/bin/kudu
/tmp/dist-test-taskup3Soe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704375975685-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704375975685-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704375975685-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704375975685-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42299
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704375975685-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704375975685-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704375975685-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704375975685-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:53:18.542622 3805 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:53:18.542899 3805 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:53:18.542964 3805 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:53:18.547348 3805 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:53:18.547417 3805 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:53:18.547513 3805 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 17:53:18.552804 3805 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704375975685-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704375975685-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704375975685-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704375975685-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42299
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3805
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704375975685-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:53:18.553906 3805 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:53:18.554919 3805 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:53:18.564116 3810 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:53:18.564086 3811 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:53:18.564078 3813 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:53:18.564508 3805 server_base.cc:1034] running on GCE node
I20260929 17:53:18.564918 3805 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:53:18.565557 3805 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:53:18.566706 3805 hybrid_clock.cc:648] HybridClock initialized: now 1790704398566679 us; error 46 us; skew 500 ppm
Sep 29 17:53:18 dist-test-slave-53nh krb5kdc[3430](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704398, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:53:18.569420 3805 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260929 17:53:18.570287 3805 webserver.cc:460] Webserver started at http://127.0.98.3:41517/ using document root <none> and password file <none>
I20260929 17:53:18.570852 3805 fs_manager.cc:359] Metadata directory not provided
I20260929 17:53:18.570922 3805 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:53:18.571117 3805 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:53:18.573150 3805 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704375975685-392-0/minicluster-data/ts-2/data/instance:
uuid: "f3e50d44bda44b61bfdb13cb43a5ce78"
format_stamp: "Formatted at 2026-09-29 17:53:18 on dist-test-slave-53nh"
server_key: "0e4c45cfc7273749a6a33ede3e79133e"
server_key_iv: "7654edc67b68d809034245b8b9cd6d85"
server_key_version: "encryptionkey@0"
I20260929 17:53:18.573695 3805 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704375975685-392-0/minicluster-data/ts-2/wal/instance:
uuid: "f3e50d44bda44b61bfdb13cb43a5ce78"
format_stamp: "Formatted at 2026-09-29 17:53:18 on dist-test-slave-53nh"
server_key: "0e4c45cfc7273749a6a33ede3e79133e"
server_key_iv: "7654edc67b68d809034245b8b9cd6d85"
server_key_version: "encryptionkey@0"
I20260929 17:53:18.577466 3805 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260929 17:53:18.579946 3820 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:53:18.581248 3805 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 17:53:18.581369 3805 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704375975685-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704375975685-392-0/minicluster-data/ts-2/wal
uuid: "f3e50d44bda44b61bfdb13cb43a5ce78"
format_stamp: "Formatted at 2026-09-29 17:53:18 on dist-test-slave-53nh"
server_key: "0e4c45cfc7273749a6a33ede3e79133e"
server_key_iv: "7654edc67b68d809034245b8b9cd6d85"
server_key_version: "encryptionkey@0"
I20260929 17:53:18.581466 3805 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704375975685-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704375975685-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704375975685-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:53:18.592587 3805 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:53:18.595587 3805 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:53:18.595752 3805 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:53:18.596312 3805 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 17:53:18.597119 3805 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:53:18.597164 3805 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:53:18.597215 3805 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:53:18.597242 3805 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 17:53:18.605185 3805 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33087
I20260929 17:53:18.605213 3933 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33087 every 8 connection(s)
I20260929 17:53:18.606031 3805 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704375975685-392-0/minicluster-data/ts-2/data/info.pb
I20260929 17:53:18.607573 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskup3Soe/build/debug/bin/kudu as pid 3805
I20260929 17:53:18.607669 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790704375975685-392-0/minicluster-data/ts-2/wal/instance
I20260929 17:53:18.607950 392 external_mini_cluster.cc:1428] Setting key 24666fe5ed0d1d638c8914f414533914
Sep 29 17:53:18 dist-test-slave-53nh krb5kdc[3430](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704398, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 17:53:18.618594 3514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:18.607714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57047 (local address 127.0.98.62:42299)
0929 17:53:18.607873 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:18.607878 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:18.608751 (+ 873us) server_negotiation.cc:407] Connection header received
0929 17:53:18.610135 (+ 1384us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:18.610137 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:18.610180 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:18.610268 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:53:18.611350 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:18.611851 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:18.612963 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:18.613248 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:18.613323 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:18.615598 (+ 2275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:53:18.615601 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:53:18.615603 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:53:18.615624 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:53:18.616723 (+ 1099us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:53:18.617205 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:53:18.617209 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:53:18.617211 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:53:18.617291 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:53:18.617579 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:53:18.617582 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:53:18.617584 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:53:18.617751 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:53:18.617832 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0929 17:53:18.618324 (+ 492us) server_negotiation.cc:299] Negotiation successful
0929 17:53:18.618435 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260929 17:53:18.619413 3936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:18.608150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42299 (local address 127.0.98.3:57047)
0929 17:53:18.608610 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:18.608662 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0929 17:53:18.609961 (+ 1299us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:53:18.610424 (+ 463us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:53:18.610431 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:53:18.610758 (+ 327us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:53:18.611245 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:18.611257 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:18.611977 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:18.611980 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:18.612865 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:18.612875 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:18.613358 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:18.613361 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:18.613511 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:18.614074 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:53:18.614089 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:53:18.615447 (+ 1358us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:53:18.616823 (+ 1376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:53:18.616828 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:53:18.616842 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:53:18.617121 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:53:18.617370 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:53:18.617374 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:53:18.617376 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:53:18.617499 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:53:18.617842 (+ 343us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:53:18.617848 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:53:18.618079 (+ 231us) client_negotiation.cc:769] Sending connection context
0929 17:53:18.618241 (+ 162us) client_negotiation.cc:240] Negotiation successful
0929 17:53:18.618548 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":110,"threads_started":1}
I20260929 17:53:18.620458 3934 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42299
I20260929 17:53:18.620676 3934 heartbeater.cc:464] Registering TS with master...
I20260929 17:53:18.621157 3934 heartbeater.cc:511] Master 127.0.98.62:42299 requested a full tablet report, sending...
I20260929 17:53:18.622077 3475 ts_manager.cc:194] Registered new tserver with Master: f3e50d44bda44b61bfdb13cb43a5ce78 (127.0.98.3:33087)
I20260929 17:53:18.622619 3475 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:57047
I20260929 17:53:18.632860 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 17:53:18 dist-test-slave-53nh krb5kdc[3430](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704398, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 29 17:53:18 dist-test-slave-53nh krb5kdc[3430](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704398, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 17:53:18.657075 3514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:18.648799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38548 (local address 127.0.98.62:42299)
0929 17:53:18.648955 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:18.648960 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:18.649117 (+ 157us) server_negotiation.cc:407] Connection header received
0929 17:53:18.649273 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:18.649276 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:18.649324 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:18.649385 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:53:18.650037 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:18.650570 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:18.651387 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:18.651603 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:18.651673 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:18.654038 (+ 2365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:53:18.654042 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:53:18.654044 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:53:18.654072 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:53:18.655421 (+ 1349us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:53:18.655788 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:53:18.655791 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:53:18.655792 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:53:18.655875 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:53:18.656215 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:53:18.656218 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:53:18.656219 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:53:18.656455 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:53:18.656558 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0929 17:53:18.656787 (+ 229us) server_negotiation.cc:299] Negotiation successful
0929 17:53:18.656885 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
W20260929 17:53:18.657509 3475 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:38548
I20260929 17:53:18.658288 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskup3Soe/build/debug/bin/kudu with pid 3537
I20260929 17:53:18.663602 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskup3Soe/build/debug/bin/kudu with pid 3671
I20260929 17:53:18.668025 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskup3Soe/build/debug/bin/kudu with pid 3805
I20260929 17:53:18.672535 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskup3Soe/build/debug/bin/kudu with pid 3443
2026-09-29T17:53:18Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2004 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704375975685-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 17:53:18 dist-test-slave-53nh krb5kdc[3946](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 17:53:18 dist-test-slave-53nh krb5kdc[3946](info): set up 2 sockets
Sep 29 17:53:18 dist-test-slave-53nh krb5kdc[3946](info): commencing operation
krb5kdc: starting...
W20260929 17:53:19.620914 392 mini_kdc.cc:121] Time spent starting KDC: real 0.929s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704375975685-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704375975685-392-0/krb5kdc/test-user.keytab.
Sep 29 17:53:19 dist-test-slave-53nh krb5kdc[3946](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704399, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T17:53:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T17:53:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704375975685-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704375975685-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 17:53:19.734058 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskup3Soe/build/debug/bin/kudu
/tmp/dist-test-taskup3Soe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704375975685-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704375975685-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704375975685-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704375975685-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37895
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704375975685-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37895
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704375975685-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704375975685-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704375975685-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:53:19.841576 3959 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:53:19.841849 3959 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:53:19.841887 3959 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:53:19.845933 3959 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 17:53:19.845995 3959 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:53:19.846014 3959 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:53:19.846027 3959 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 17:53:19.846041 3959 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 17:53:19.851327 3959 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704375975685-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704375975685-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37895
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704375975685-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37895
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704375975685-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3959
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704375975685-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:53:19.852349 3959 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:53:19.853374 3959 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 17:53:19.861434 3959 server_base.cc:1034] running on GCE node
W20260929 17:53:19.861383 3964 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:53:19.861387 3965 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:53:19.861383 3967 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:53:19.862110 3959 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:53:19.863168 3959 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:53:19.864349 3959 hybrid_clock.cc:648] HybridClock initialized: now 1790704399864330 us; error 31 us; skew 500 ppm
Sep 29 17:53:19 dist-test-slave-53nh krb5kdc[3946](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704399, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:53:19.866935 3959 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 17:53:19.867753 3959 webserver.cc:460] Webserver started at http://127.0.98.62:39323/ using document root <none> and password file <none>
I20260929 17:53:19.868244 3959 fs_manager.cc:359] Metadata directory not provided
I20260929 17:53:19.868299 3959 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:53:19.868474 3959 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:53:19.870438 3959 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704375975685-392-0/minicluster-data/master-0/data/instance:
uuid: "bf8942fbf1c14430b44616df2c01d948"
format_stamp: "Formatted at 2026-09-29 17:53:19 on dist-test-slave-53nh"
server_key: "6f88713119a286a2ae749565739423a6"
server_key_iv: "c52ad229796e9313c82c8f526cd89e75"
server_key_version: "encryptionkey@0"
I20260929 17:53:19.870926 3959 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704375975685-392-0/minicluster-data/master-0/wal/instance:
uuid: "bf8942fbf1c14430b44616df2c01d948"
format_stamp: "Formatted at 2026-09-29 17:53:19 on dist-test-slave-53nh"
server_key: "6f88713119a286a2ae749565739423a6"
server_key_iv: "c52ad229796e9313c82c8f526cd89e75"
server_key_version: "encryptionkey@0"
I20260929 17:53:19.874351 3959 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 17:53:19.876577 3974 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:53:19.877753 3959 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 17:53:19.877846 3959 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704375975685-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704375975685-392-0/minicluster-data/master-0/wal
uuid: "bf8942fbf1c14430b44616df2c01d948"
format_stamp: "Formatted at 2026-09-29 17:53:19 on dist-test-slave-53nh"
server_key: "6f88713119a286a2ae749565739423a6"
server_key_iv: "c52ad229796e9313c82c8f526cd89e75"
server_key_version: "encryptionkey@0"
I20260929 17:53:19.877925 3959 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704375975685-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704375975685-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704375975685-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:53:19.902400 3959 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:53:19.905283 3959 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:53:19.905433 3959 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:53:19.911983 3959 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37895
I20260929 17:53:19.912004 4026 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37895 every 8 connection(s)
I20260929 17:53:19.912914 3959 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704375975685-392-0/minicluster-data/master-0/data/info.pb
I20260929 17:53:19.915583 4027 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:53:19.919936 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskup3Soe/build/debug/bin/kudu as pid 3959
I20260929 17:53:19.920059 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704375975685-392-0/minicluster-data/master-0/wal/instance
I20260929 17:53:19.920342 392 external_mini_cluster.cc:1428] Setting key 45a25b1b3388ac88845ebf4f59be098c
I20260929 17:53:19.921187 4027 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf8942fbf1c14430b44616df2c01d948: Bootstrap starting.
I20260929 17:53:19.923404 4027 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bf8942fbf1c14430b44616df2c01d948: Neither blocks nor log segments found. Creating new log.
I20260929 17:53:19.924269 4027 log.cc:824] T 00000000000000000000000000000000 P bf8942fbf1c14430b44616df2c01d948: Log is configured to *not* fsync() on all Append() calls
I20260929 17:53:19.926282 4027 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf8942fbf1c14430b44616df2c01d948: No bootstrap required, opened a new log
Sep 29 17:53:19 dist-test-slave-53nh krb5kdc[3946](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704399, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 17:53:19.930853 4027 raft_consensus.cc:348] T 00000000000000000000000000000000 P bf8942fbf1c14430b44616df2c01d948 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf8942fbf1c14430b44616df2c01d948" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37895 } }
I20260929 17:53:19.931111 4027 raft_consensus.cc:374] T 00000000000000000000000000000000 P bf8942fbf1c14430b44616df2c01d948 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:53:19.931166 4027 raft_consensus.cc:729] T 00000000000000000000000000000000 P bf8942fbf1c14430b44616df2c01d948 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf8942fbf1c14430b44616df2c01d948, State: Initialized, Role: FOLLOWER
I20260929 17:53:19.931635 4027 consensus_queue.cc:260] T 00000000000000000000000000000000 P bf8942fbf1c14430b44616df2c01d948 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf8942fbf1c14430b44616df2c01d948" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37895 } }
I20260929 17:53:19.931768 4027 raft_consensus.cc:388] T 00000000000000000000000000000000 P bf8942fbf1c14430b44616df2c01d948 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 17:53:19.931828 4027 raft_consensus.cc:482] T 00000000000000000000000000000000 P bf8942fbf1c14430b44616df2c01d948 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 17:53:19.931910 4027 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bf8942fbf1c14430b44616df2c01d948 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:53:19.932739 4027 raft_consensus.cc:504] T 00000000000000000000000000000000 P bf8942fbf1c14430b44616df2c01d948 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf8942fbf1c14430b44616df2c01d948" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37895 } }
I20260929 17:53:19.933084 4027 leader_election.cc:302] T 00000000000000000000000000000000 P bf8942fbf1c14430b44616df2c01d948 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bf8942fbf1c14430b44616df2c01d948; no voters:
I20260929 17:53:19.933411 4027 leader_election.cc:288] T 00000000000000000000000000000000 P bf8942fbf1c14430b44616df2c01d948 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 17:53:19.933570 4032 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bf8942fbf1c14430b44616df2c01d948 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 17:53:19.933737 4032 raft_consensus.cc:686] T 00000000000000000000000000000000 P bf8942fbf1c14430b44616df2c01d948 [term 1 LEADER]: Becoming Leader. State: Replica: bf8942fbf1c14430b44616df2c01d948, State: Running, Role: LEADER
I20260929 17:53:19.934018 4032 consensus_queue.cc:237] T 00000000000000000000000000000000 P bf8942fbf1c14430b44616df2c01d948 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf8942fbf1c14430b44616df2c01d948" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37895 } }
I20260929 17:53:19.934441 4030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:19.921668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55966 (local address 127.0.98.62:37895)
0929 17:53:19.922019 (+ 351us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:19.922034 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:19.922067 (+ 33us) server_negotiation.cc:407] Connection header received
0929 17:53:19.922656 (+ 589us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:19.922672 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:19.922996 (+ 324us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:19.923262 (+ 266us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:53:19.924111 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:19.925265 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:19.926355 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:19.926822 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:19.926932 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:19.929544 (+ 2612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:53:19.929551 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:53:19.929568 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:53:19.929600 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:53:19.931425 (+ 1825us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:53:19.931851 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:53:19.931857 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:53:19.931864 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:53:19.932013 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:53:19.932316 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:53:19.932319 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:53:19.932321 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:53:19.932720 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:53:19.932901 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0929 17:53:19.933217 (+ 316us) server_negotiation.cc:299] Negotiation successful
0929 17:53:19.933532 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":224,"thread_start_us":87,"threads_started":1}
I20260929 17:53:19.934700 4027 sys_catalog.cc:564] T 00000000000000000000000000000000 P bf8942fbf1c14430b44616df2c01d948 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 17:53:19.935626 4034 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf8942fbf1c14430b44616df2c01d948 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bf8942fbf1c14430b44616df2c01d948. Latest consensus state: current_term: 1 leader_uuid: "bf8942fbf1c14430b44616df2c01d948" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf8942fbf1c14430b44616df2c01d948" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37895 } } }
I20260929 17:53:19.935747 4034 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf8942fbf1c14430b44616df2c01d948 [sys.catalog]: This master's current role is: LEADER
I20260929 17:53:19.935722 4033 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf8942fbf1c14430b44616df2c01d948 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bf8942fbf1c14430b44616df2c01d948" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf8942fbf1c14430b44616df2c01d948" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37895 } } }
I20260929 17:53:19.935786 4033 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf8942fbf1c14430b44616df2c01d948 [sys.catalog]: This master's current role is: LEADER
I20260929 17:53:19.936569 4041 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 17:53:19.939608 4041 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 17:53:19.945360 4041 catalog_manager.cc:1347] Generated new cluster ID: 9d13900141b04140aeec703236e52b70
I20260929 17:53:19.945418 4041 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 17:53:19.951332 4041 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 17:53:19.952179 4041 catalog_manager.cc:1504] Loading token signing keys...
I20260929 17:53:19.957038 4041 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bf8942fbf1c14430b44616df2c01d948: Generated new TSK 0
I20260929 17:53:19.957630 4041 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704375975685-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704375975685-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260929 17:53:19.987869 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskup3Soe/build/debug/bin/kudu
/tmp/dist-test-taskup3Soe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704375975685-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704375975685-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704375975685-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704375975685-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37895
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704375975685-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704375975685-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704375975685-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704375975685-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:53:20.121146 4053 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:53:20.121440 4053 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:53:20.121500 4053 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:53:20.125669 4053 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:53:20.125735 4053 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:53:20.125820 4053 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 17:53:20.131107 4053 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704375975685-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704375975685-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704375975685-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704375975685-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37895
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4053
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704375975685-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:53:20.132120 4053 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:53:20.133152 4053 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:53:20.142315 4058 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:53:20.142323 4059 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:53:20.142309 4061 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:53:20.142481 4053 server_base.cc:1034] running on GCE node
I20260929 17:53:20.142930 4053 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:53:20.143537 4053 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:53:20.144685 4053 hybrid_clock.cc:648] HybridClock initialized: now 1790704400144667 us; error 29 us; skew 500 ppm
Sep 29 17:53:20 dist-test-slave-53nh krb5kdc[3946](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704400, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:53:20.147576 4053 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260929 17:53:20.148550 4053 webserver.cc:460] Webserver started at http://127.0.98.1:40297/ using document root <none> and password file <none>
I20260929 17:53:20.149138 4053 fs_manager.cc:359] Metadata directory not provided
I20260929 17:53:20.149210 4053 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:53:20.149410 4053 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:53:20.151397 4053 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704375975685-392-0/minicluster-data/ts-0/data/instance:
uuid: "0b82d07400b24b2ba129acd0b8c1b595"
format_stamp: "Formatted at 2026-09-29 17:53:20 on dist-test-slave-53nh"
server_key: "f49043325fa53ab37675ae536331e69c"
server_key_iv: "fa9e38902312e4cbdb3ad95a73ba8f4b"
server_key_version: "encryptionkey@0"
I20260929 17:53:20.151904 4053 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704375975685-392-0/minicluster-data/ts-0/wal/instance:
uuid: "0b82d07400b24b2ba129acd0b8c1b595"
format_stamp: "Formatted at 2026-09-29 17:53:20 on dist-test-slave-53nh"
server_key: "f49043325fa53ab37675ae536331e69c"
server_key_iv: "fa9e38902312e4cbdb3ad95a73ba8f4b"
server_key_version: "encryptionkey@0"
I20260929 17:53:20.155426 4053 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 17:53:20.157738 4068 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:53:20.158955 4053 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 17:53:20.159070 4053 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704375975685-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704375975685-392-0/minicluster-data/ts-0/wal
uuid: "0b82d07400b24b2ba129acd0b8c1b595"
format_stamp: "Formatted at 2026-09-29 17:53:20 on dist-test-slave-53nh"
server_key: "f49043325fa53ab37675ae536331e69c"
server_key_iv: "fa9e38902312e4cbdb3ad95a73ba8f4b"
server_key_version: "encryptionkey@0"
I20260929 17:53:20.159163 4053 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704375975685-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704375975685-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704375975685-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:53:20.181592 4053 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:53:20.184875 4053 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:53:20.185072 4053 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:53:20.185645 4053 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 17:53:20.186450 4053 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:53:20.186496 4053 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:53:20.186547 4053 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:53:20.186575 4053 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 17:53:20.194794 4053 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36983
I20260929 17:53:20.194835 4181 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36983 every 8 connection(s)
I20260929 17:53:20.195662 4053 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704375975685-392-0/minicluster-data/ts-0/data/info.pb
I20260929 17:53:20.204506 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskup3Soe/build/debug/bin/kudu as pid 4053
I20260929 17:53:20.204623 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704375975685-392-0/minicluster-data/ts-0/wal/instance
I20260929 17:53:20.204942 392 external_mini_cluster.cc:1428] Setting key deba6918758f10995c5f8479491bccb6
Sep 29 17:53:20 dist-test-slave-53nh krb5kdc[3946](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704400, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 17:53:20.209158 4030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:20.197428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50401 (local address 127.0.98.62:37895)
0929 17:53:20.197552 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:20.197555 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:20.198413 (+ 858us) server_negotiation.cc:407] Connection header received
0929 17:53:20.199768 (+ 1355us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:20.199772 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:20.199844 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:20.199914 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:53:20.201116 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:20.201660 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:20.202791 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:20.203028 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:20.203118 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:20.205601 (+ 2483us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:53:20.205604 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:53:20.205606 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:53:20.205633 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:53:20.207071 (+ 1438us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:53:20.207632 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:53:20.207635 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:53:20.207636 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:53:20.207731 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:53:20.208077 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:53:20.208080 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:53:20.208082 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:53:20.208297 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:53:20.208418 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0929 17:53:20.208852 (+ 434us) server_negotiation.cc:299] Negotiation successful
0929 17:53:20.208973 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 17:53:20.209949 4184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:20.197822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37895 (local address 127.0.98.1:50401)
0929 17:53:20.198261 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:20.198311 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0929 17:53:20.199564 (+ 1253us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:53:20.200098 (+ 534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:53:20.200105 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:53:20.200461 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:53:20.200929 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:20.200944 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:20.201772 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:20.201774 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:20.202664 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:20.202673 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:20.203150 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:20.203152 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:20.203307 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:20.203827 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:53:20.203842 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:53:20.205437 (+ 1595us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:53:20.207180 (+ 1743us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:53:20.207186 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:53:20.207202 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:53:20.207524 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:53:20.207834 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:53:20.207837 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:53:20.207839 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:53:20.207998 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:53:20.208409 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:53:20.208413 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:53:20.208638 (+ 225us) client_negotiation.cc:769] Sending connection context
0929 17:53:20.208797 (+ 159us) client_negotiation.cc:240] Negotiation successful
0929 17:53:20.209087 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":114,"threads_started":1}
I20260929 17:53:20.211102 4182 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37895
I20260929 17:53:20.211375 4182 heartbeater.cc:464] Registering TS with master...
I20260929 17:53:20.211949 4182 heartbeater.cc:511] Master 127.0.98.62:37895 requested a full tablet report, sending...
I20260929 17:53:20.213480 3991 ts_manager.cc:194] Registered new tserver with Master: 0b82d07400b24b2ba129acd0b8c1b595 (127.0.98.1:36983)
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260929 17:53:20.214586 3991 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:50401
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704375975685-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704375975685-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260929 17:53:20.231392 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskup3Soe/build/debug/bin/kudu
/tmp/dist-test-taskup3Soe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704375975685-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704375975685-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704375975685-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704375975685-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37895
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704375975685-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704375975685-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704375975685-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704375975685-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:53:20.341249 4187 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:53:20.341500 4187 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:53:20.341544 4187 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:53:20.345853 4187 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:53:20.345918 4187 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:53:20.345989 4187 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 17:53:20.351466 4187 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704375975685-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704375975685-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704375975685-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704375975685-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37895
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4187
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704375975685-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:53:20.352512 4187 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:53:20.353593 4187 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:53:20.362891 4192 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:53:20.362906 4195 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:53:20.362980 4187 server_base.cc:1034] running on GCE node
W20260929 17:53:20.362906 4193 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:53:20.363499 4187 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:53:20.364144 4187 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:53:20.365293 4187 hybrid_clock.cc:648] HybridClock initialized: now 1790704400365274 us; error 31 us; skew 500 ppm
Sep 29 17:53:20 dist-test-slave-53nh krb5kdc[3946](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704400, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:53:20.367962 4187 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260929 17:53:20.368819 4187 webserver.cc:460] Webserver started at http://127.0.98.2:40519/ using document root <none> and password file <none>
I20260929 17:53:20.369391 4187 fs_manager.cc:359] Metadata directory not provided
I20260929 17:53:20.369451 4187 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:53:20.369622 4187 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:53:20.371568 4187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704375975685-392-0/minicluster-data/ts-1/data/instance:
uuid: "1283cbe00bf546998b5e3ba8e55b6f40"
format_stamp: "Formatted at 2026-09-29 17:53:20 on dist-test-slave-53nh"
server_key: "e67b964347ca681843043e3ac36010ab"
server_key_iv: "085b39d2a5030a2ad86b8e196e3e2857"
server_key_version: "encryptionkey@0"
I20260929 17:53:20.372078 4187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704375975685-392-0/minicluster-data/ts-1/wal/instance:
uuid: "1283cbe00bf546998b5e3ba8e55b6f40"
format_stamp: "Formatted at 2026-09-29 17:53:20 on dist-test-slave-53nh"
server_key: "e67b964347ca681843043e3ac36010ab"
server_key_iv: "085b39d2a5030a2ad86b8e196e3e2857"
server_key_version: "encryptionkey@0"
I20260929 17:53:20.375593 4187 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260929 17:53:20.377913 4202 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:53:20.379109 4187 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 17:53:20.379209 4187 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704375975685-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704375975685-392-0/minicluster-data/ts-1/wal
uuid: "1283cbe00bf546998b5e3ba8e55b6f40"
format_stamp: "Formatted at 2026-09-29 17:53:20 on dist-test-slave-53nh"
server_key: "e67b964347ca681843043e3ac36010ab"
server_key_iv: "085b39d2a5030a2ad86b8e196e3e2857"
server_key_version: "encryptionkey@0"
I20260929 17:53:20.379316 4187 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704375975685-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704375975685-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704375975685-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:53:20.400063 4187 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:53:20.403308 4187 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:53:20.403482 4187 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:53:20.404110 4187 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 17:53:20.404953 4187 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:53:20.405021 4187 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:53:20.405083 4187 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:53:20.405112 4187 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 17:53:20.413575 4187 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44919
I20260929 17:53:20.413611 4315 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44919 every 8 connection(s)
I20260929 17:53:20.414455 4187 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704375975685-392-0/minicluster-data/ts-1/data/info.pb
I20260929 17:53:20.417621 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskup3Soe/build/debug/bin/kudu as pid 4187
I20260929 17:53:20.417742 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704375975685-392-0/minicluster-data/ts-1/wal/instance
I20260929 17:53:20.417987 392 external_mini_cluster.cc:1428] Setting key cc51bc696de04232692e1410e94a3a81
Sep 29 17:53:20 dist-test-slave-53nh krb5kdc[3946](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704400, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260929 17:53:20.428341 4030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:20.416225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40151 (local address 127.0.98.62:37895)
0929 17:53:20.416357 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:20.416360 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:20.417206 (+ 846us) server_negotiation.cc:407] Connection header received
0929 17:53:20.418602 (+ 1396us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:20.418606 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:20.418663 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:20.418735 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:53:20.419902 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:20.420419 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:20.421620 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:20.421839 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:20.421931 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:20.424513 (+ 2582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:53:20.424516 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:53:20.424518 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:53:20.424541 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:53:20.426092 (+ 1551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:53:20.426640 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:53:20.426644 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:53:20.426646 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:53:20.426730 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:53:20.427103 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:53:20.427107 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:53:20.427109 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:53:20.427325 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:53:20.427412 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0929 17:53:20.427983 (+ 571us) server_negotiation.cc:299] Negotiation successful
0929 17:53:20.428109 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":24,"server-negotiator.queue_time_us":47}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260929 17:53:20.429198 4318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:20.416554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37895 (local address 127.0.98.2:40151)
0929 17:53:20.417049 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:20.417105 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0929 17:53:20.418394 (+ 1289us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:53:20.418929 (+ 535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:53:20.418937 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:53:20.419281 (+ 344us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:53:20.419760 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:20.419772 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:20.420537 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:20.420540 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:20.421503 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:20.421513 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:20.422010 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:20.422014 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:20.422183 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:20.422938 (+ 755us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:53:20.422957 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:53:20.424340 (+ 1383us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:53:20.426201 (+ 1861us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:53:20.426207 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:53:20.426224 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:53:20.426540 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:53:20.426834 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:53:20.426837 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:53:20.426838 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:53:20.427019 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:53:20.427427 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:53:20.427433 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:53:20.427711 (+ 278us) client_negotiation.cc:769] Sending connection context
0929 17:53:20.427933 (+ 222us) client_negotiation.cc:240] Negotiation successful
0929 17:53:20.428273 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":113,"threads_started":1}
I20260929 17:53:20.430325 4316 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37895
I20260929 17:53:20.430603 4316 heartbeater.cc:464] Registering TS with master...
I20260929 17:53:20.431188 4316 heartbeater.cc:511] Master 127.0.98.62:37895 requested a full tablet report, sending...
I20260929 17:53:20.432284 3991 ts_manager.cc:194] Registered new tserver with Master: 1283cbe00bf546998b5e3ba8e55b6f40 (127.0.98.2:44919)
I20260929 17:53:20.432894 3991 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:40151
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704375975685-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704375975685-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260929 17:53:20.443516 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskup3Soe/build/debug/bin/kudu
/tmp/dist-test-taskup3Soe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704375975685-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704375975685-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704375975685-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704375975685-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37895
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704375975685-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704375975685-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704375975685-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704375975685-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:53:20.552095 4321 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:53:20.552384 4321 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:53:20.552428 4321 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:53:20.556547 4321 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:53:20.556630 4321 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:53:20.556690 4321 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 17:53:20.562705 4321 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704375975685-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704375975685-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704375975685-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704375975685-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37895
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4321
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704375975685-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:53:20.563795 4321 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:53:20.564834 4321 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:53:20.574373 4329 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:53:20.574414 4321 server_base.cc:1034] running on GCE node
W20260929 17:53:20.574401 4326 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:53:20.574394 4327 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:53:20.575030 4321 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:53:20.575655 4321 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:53:20.576798 4321 hybrid_clock.cc:648] HybridClock initialized: now 1790704400576774 us; error 37 us; skew 500 ppm
Sep 29 17:53:20 dist-test-slave-53nh krb5kdc[3946](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704400, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:53:20.579478 4321 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260929 17:53:20.580346 4321 webserver.cc:460] Webserver started at http://127.0.98.3:37657/ using document root <none> and password file <none>
I20260929 17:53:20.580884 4321 fs_manager.cc:359] Metadata directory not provided
I20260929 17:53:20.580948 4321 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:53:20.581203 4321 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:53:20.583467 4321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704375975685-392-0/minicluster-data/ts-2/data/instance:
uuid: "9ffd1a71ac154b42a8b53a58626e84db"
format_stamp: "Formatted at 2026-09-29 17:53:20 on dist-test-slave-53nh"
server_key: "720d7014c0158b239d9e626ae96bda96"
server_key_iv: "523da9d53658dc3614193e789cee86e6"
server_key_version: "encryptionkey@0"
I20260929 17:53:20.584028 4321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704375975685-392-0/minicluster-data/ts-2/wal/instance:
uuid: "9ffd1a71ac154b42a8b53a58626e84db"
format_stamp: "Formatted at 2026-09-29 17:53:20 on dist-test-slave-53nh"
server_key: "720d7014c0158b239d9e626ae96bda96"
server_key_iv: "523da9d53658dc3614193e789cee86e6"
server_key_version: "encryptionkey@0"
I20260929 17:53:20.587702 4321 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260929 17:53:20.590096 4336 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:53:20.591256 4321 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 17:53:20.591348 4321 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704375975685-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704375975685-392-0/minicluster-data/ts-2/wal
uuid: "9ffd1a71ac154b42a8b53a58626e84db"
format_stamp: "Formatted at 2026-09-29 17:53:20 on dist-test-slave-53nh"
server_key: "720d7014c0158b239d9e626ae96bda96"
server_key_iv: "523da9d53658dc3614193e789cee86e6"
server_key_version: "encryptionkey@0"
I20260929 17:53:20.591444 4321 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704375975685-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704375975685-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704375975685-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:53:20.602355 4321 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:53:20.605357 4321 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:53:20.605518 4321 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:53:20.606062 4321 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 17:53:20.606827 4321 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:53:20.606870 4321 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:53:20.606958 4321 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:53:20.606986 4321 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 17:53:20.614966 4321 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41753
I20260929 17:53:20.614993 4449 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41753 every 8 connection(s)
I20260929 17:53:20.615799 4321 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704375975685-392-0/minicluster-data/ts-2/data/info.pb
I20260929 17:53:20.619410 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskup3Soe/build/debug/bin/kudu as pid 4321
I20260929 17:53:20.619495 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790704375975685-392-0/minicluster-data/ts-2/wal/instance
I20260929 17:53:20.619740 392 external_mini_cluster.cc:1428] Setting key 58275a3eea3fa109b7b44840c341f0bc
Sep 29 17:53:20 dist-test-slave-53nh krb5kdc[3946](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704400, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 17:53:20.628830 4030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:20.617575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39053 (local address 127.0.98.62:37895)
0929 17:53:20.617719 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:20.617723 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:20.618525 (+ 802us) server_negotiation.cc:407] Connection header received
0929 17:53:20.619902 (+ 1377us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:20.619906 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:20.619961 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:20.620046 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:53:20.621444 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:20.622084 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:20.623177 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:20.623397 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:20.623468 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:20.625705 (+ 2237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:53:20.625708 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:53:20.625710 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:53:20.625750 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:53:20.627027 (+ 1277us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:53:20.627497 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:53:20.627499 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:53:20.627500 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:53:20.627564 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:53:20.627893 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:53:20.627896 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:53:20.627898 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:53:20.628084 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:53:20.628156 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0929 17:53:20.628607 (+ 451us) server_negotiation.cc:299] Negotiation successful
0929 17:53:20.628694 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 17:53:20.629642 4452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:20.617916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37895 (local address 127.0.98.3:39053)
0929 17:53:20.618360 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:20.618413 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0929 17:53:20.619675 (+ 1262us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:53:20.620235 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:53:20.620244 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:53:20.620686 (+ 442us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:53:20.621298 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:20.621313 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:20.622189 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:20.622192 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:20.623082 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:20.623091 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:20.623505 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:20.623507 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:20.623642 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:20.624184 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:53:20.624199 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:53:20.625556 (+ 1357us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:53:20.627108 (+ 1552us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:53:20.627111 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:53:20.627122 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:53:20.627424 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:53:20.627630 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:53:20.627633 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:53:20.627635 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:53:20.627794 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:53:20.628175 (+ 381us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:53:20.628180 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:53:20.628425 (+ 245us) client_negotiation.cc:769] Sending connection context
0929 17:53:20.628578 (+ 153us) client_negotiation.cc:240] Negotiation successful
0929 17:53:20.628821 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":106,"threads_started":1}
I20260929 17:53:20.630635 4450 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37895
I20260929 17:53:20.630838 4450 heartbeater.cc:464] Registering TS with master...
I20260929 17:53:20.631261 4450 heartbeater.cc:511] Master 127.0.98.62:37895 requested a full tablet report, sending...
I20260929 17:53:20.632181 3991 ts_manager.cc:194] Registered new tserver with Master: 9ffd1a71ac154b42a8b53a58626e84db (127.0.98.3:41753)
I20260929 17:53:20.632752 3991 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:39053
I20260929 17:53:20.633498 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 17:53:20 dist-test-slave-53nh krb5kdc[3946](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704400, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 17:53:20 dist-test-slave-53nh krb5kdc[3946](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704400, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260929 17:53:20.656896 4457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:20.648283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50934 (local address 127.0.98.1:36983)
0929 17:53:20.648623 (+ 340us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:20.648631 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:20.648653 (+ 22us) server_negotiation.cc:407] Connection header received
0929 17:53:20.648704 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:20.648709 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:20.648916 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:20.649064 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:53:20.649682 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:20.650438 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:20.651190 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:20.651553 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:20.651631 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:20.653941 (+ 2310us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:53:20.653949 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:53:20.653963 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:53:20.653986 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:53:20.655469 (+ 1483us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:53:20.655832 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:53:20.655838 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:53:20.655839 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:53:20.655910 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:53:20.656088 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:53:20.656090 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:53:20.656091 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:53:20.656328 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:53:20.656486 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0929 17:53:20.656622 (+ 136us) server_negotiation.cc:299] Negotiation successful
0929 17:53:20.656741 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":244,"thread_start_us":146,"threads_started":1}
Sep 29 17:53:20 dist-test-slave-53nh krb5kdc[3946](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704400, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 17:53:20.666698 4030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:20.659681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55980 (local address 127.0.98.62:37895)
0929 17:53:20.659816 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:20.659820 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:20.659836 (+ 16us) server_negotiation.cc:407] Connection header received
0929 17:53:20.659929 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:20.659932 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:20.659969 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:20.660031 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:53:20.660530 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:20.661061 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:20.661769 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:20.661970 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:20.662036 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:20.664150 (+ 2114us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:53:20.664153 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:53:20.664155 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:53:20.664177 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:53:20.665454 (+ 1277us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:53:20.665775 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:53:20.665777 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:53:20.665778 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:53:20.665844 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:53:20.666025 (+ 181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:53:20.666027 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:53:20.666028 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:53:20.666224 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:53:20.666318 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0929 17:53:20.666488 (+ 170us) server_negotiation.cc:299] Negotiation successful
0929 17:53:20.666577 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
W20260929 17:53:20.667075 3991 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55980
I20260929 17:53:20.667639 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskup3Soe/build/debug/bin/kudu with pid 4053
I20260929 17:53:20.672607 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskup3Soe/build/debug/bin/kudu with pid 4187
I20260929 17:53:20.677354 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskup3Soe/build/debug/bin/kudu with pid 4321
I20260929 17:53:20.681610 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskup3Soe/build/debug/bin/kudu with pid 3959
2026-09-29T17:53:20Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2009 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704375975685-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 17:53:20 dist-test-slave-53nh krb5kdc[4462](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 17:53:20 dist-test-slave-53nh krb5kdc[4462](info): set up 2 sockets
Sep 29 17:53:20 dist-test-slave-53nh krb5kdc[4462](info): commencing operation
krb5kdc: starting...
W20260929 17:53:21.624971 392 mini_kdc.cc:121] Time spent starting KDC: real 0.924s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704375975685-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704375975685-392-0/krb5kdc/test-user.keytab.
Sep 29 17:53:21 dist-test-slave-53nh krb5kdc[4462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704401, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T17:53:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T17:53:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704375975685-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704375975685-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 17:53:21.739295 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskup3Soe/build/debug/bin/kudu
/tmp/dist-test-taskup3Soe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704375975685-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704375975685-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704375975685-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704375975685-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45271
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704375975685-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45271
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704375975685-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704375975685-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704375975685-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:53:21.845896 4475 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:53:21.846205 4475 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:53:21.846277 4475 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:53:21.850749 4475 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 17:53:21.850843 4475 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:53:21.850872 4475 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:53:21.850895 4475 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 17:53:21.850924 4475 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 17:53:21.858117 4475 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704375975685-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704375975685-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45271
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704375975685-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45271
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704375975685-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4475
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704375975685-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:53:21.859645 4475 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:53:21.861060 4475 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:53:21.871028 4483 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:53:21.871032 4480 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:53:21.871299 4475 server_base.cc:1034] running on GCE node
W20260929 17:53:21.871416 4481 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:53:21.872020 4475 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:53:21.873497 4475 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:53:21.874686 4475 hybrid_clock.cc:648] HybridClock initialized: now 1790704401874666 us; error 33 us; skew 500 ppm
Sep 29 17:53:21 dist-test-slave-53nh krb5kdc[4462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704401, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:53:21.878082 4475 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 17:53:21.878898 4475 fs_manager.cc:359] Metadata directory not provided
I20260929 17:53:21.878978 4475 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:53:21.879195 4475 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:53:21.881791 4475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704375975685-392-0/minicluster-data/master-0/data/instance:
uuid: "f6966161f41a458e91ad757e8ac08b49"
format_stamp: "Formatted at 2026-09-29 17:53:21 on dist-test-slave-53nh"
server_key: "78cdbfea93e5aac57088c2a0240949b5"
server_key_iv: "6cb78c4a83509d3dd8b5ef91ea4e5151"
server_key_version: "encryptionkey@0"
I20260929 17:53:21.882373 4475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704375975685-392-0/minicluster-data/master-0/wal/instance:
uuid: "f6966161f41a458e91ad757e8ac08b49"
format_stamp: "Formatted at 2026-09-29 17:53:21 on dist-test-slave-53nh"
server_key: "78cdbfea93e5aac57088c2a0240949b5"
server_key_iv: "6cb78c4a83509d3dd8b5ef91ea4e5151"
server_key_version: "encryptionkey@0"
I20260929 17:53:21.889842 4475 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.001s sys 0.004s
I20260929 17:53:21.892863 4489 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:53:21.894505 4475 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 17:53:21.894647 4475 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704375975685-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704375975685-392-0/minicluster-data/master-0/wal
uuid: "f6966161f41a458e91ad757e8ac08b49"
format_stamp: "Formatted at 2026-09-29 17:53:21 on dist-test-slave-53nh"
server_key: "78cdbfea93e5aac57088c2a0240949b5"
server_key_iv: "6cb78c4a83509d3dd8b5ef91ea4e5151"
server_key_version: "encryptionkey@0"
I20260929 17:53:21.894779 4475 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704375975685-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704375975685-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704375975685-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:53:21.909701 4475 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:53:21.912817 4475 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:53:21.912992 4475 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:53:21.919752 4475 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45271
I20260929 17:53:21.920001 4541 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45271 every 8 connection(s)
I20260929 17:53:21.920389 4475 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704375975685-392-0/minicluster-data/master-0/data/info.pb
I20260929 17:53:21.923161 4542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:53:21.925655 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskup3Soe/build/debug/bin/kudu as pid 4475
I20260929 17:53:21.925791 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704375975685-392-0/minicluster-data/master-0/wal/instance
I20260929 17:53:21.926117 392 external_mini_cluster.cc:1428] Setting key 52e795c0b9cf80ef5aa2e88a0e23639f
I20260929 17:53:21.929519 4542 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f6966161f41a458e91ad757e8ac08b49: Bootstrap starting.
I20260929 17:53:21.932075 4542 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f6966161f41a458e91ad757e8ac08b49: Neither blocks nor log segments found. Creating new log.
I20260929 17:53:21.932811 4542 log.cc:824] T 00000000000000000000000000000000 P f6966161f41a458e91ad757e8ac08b49: Log is configured to *not* fsync() on all Append() calls
I20260929 17:53:21.935025 4542 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f6966161f41a458e91ad757e8ac08b49: No bootstrap required, opened a new log
Sep 29 17:53:21 dist-test-slave-53nh krb5kdc[4462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704401, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 17:53:21.938088 4542 raft_consensus.cc:348] T 00000000000000000000000000000000 P f6966161f41a458e91ad757e8ac08b49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6966161f41a458e91ad757e8ac08b49" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45271 } }
I20260929 17:53:21.938318 4542 raft_consensus.cc:374] T 00000000000000000000000000000000 P f6966161f41a458e91ad757e8ac08b49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:53:21.938386 4542 raft_consensus.cc:729] T 00000000000000000000000000000000 P f6966161f41a458e91ad757e8ac08b49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6966161f41a458e91ad757e8ac08b49, State: Initialized, Role: FOLLOWER
I20260929 17:53:21.938872 4542 consensus_queue.cc:260] T 00000000000000000000000000000000 P f6966161f41a458e91ad757e8ac08b49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6966161f41a458e91ad757e8ac08b49" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45271 } }
I20260929 17:53:21.939045 4542 raft_consensus.cc:388] T 00000000000000000000000000000000 P f6966161f41a458e91ad757e8ac08b49 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 17:53:21.939129 4542 raft_consensus.cc:482] T 00000000000000000000000000000000 P f6966161f41a458e91ad757e8ac08b49 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 17:53:21.939225 4542 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f6966161f41a458e91ad757e8ac08b49 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:53:21.940375 4542 raft_consensus.cc:504] T 00000000000000000000000000000000 P f6966161f41a458e91ad757e8ac08b49 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6966161f41a458e91ad757e8ac08b49" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45271 } }
I20260929 17:53:21.940752 4542 leader_election.cc:302] T 00000000000000000000000000000000 P f6966161f41a458e91ad757e8ac08b49 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f6966161f41a458e91ad757e8ac08b49; no voters:
I20260929 17:53:21.941145 4542 leader_election.cc:288] T 00000000000000000000000000000000 P f6966161f41a458e91ad757e8ac08b49 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 17:53:21.941217 4545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:21.927640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53602 (local address 127.0.98.62:45271)
0929 17:53:21.928089 (+ 449us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:21.928105 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:21.928181 (+ 76us) server_negotiation.cc:407] Connection header received
0929 17:53:21.928816 (+ 635us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:21.928833 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:21.929243 (+ 410us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:21.929566 (+ 323us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:53:21.930321 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:21.931666 (+ 1345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:21.932559 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:21.933080 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:21.933246 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:21.935942 (+ 2696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:53:21.935947 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:53:21.935960 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:53:21.935987 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:53:21.938019 (+ 2032us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:53:21.938444 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:53:21.938451 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:53:21.938457 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:53:21.938618 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:53:21.938885 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:53:21.938888 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:53:21.938890 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:53:21.939426 (+ 536us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:53:21.939639 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0929 17:53:21.939842 (+ 203us) server_negotiation.cc:299] Negotiation successful
0929 17:53:21.940151 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":303,"thread_start_us":131,"threads_started":1}
I20260929 17:53:21.941231 4547 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f6966161f41a458e91ad757e8ac08b49 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 17:53:21.941457 4547 raft_consensus.cc:686] T 00000000000000000000000000000000 P f6966161f41a458e91ad757e8ac08b49 [term 1 LEADER]: Becoming Leader. State: Replica: f6966161f41a458e91ad757e8ac08b49, State: Running, Role: LEADER
I20260929 17:53:21.941767 4547 consensus_queue.cc:237] T 00000000000000000000000000000000 P f6966161f41a458e91ad757e8ac08b49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6966161f41a458e91ad757e8ac08b49" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45271 } }
I20260929 17:53:21.942289 4542 sys_catalog.cc:564] T 00000000000000000000000000000000 P f6966161f41a458e91ad757e8ac08b49 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 17:53:21.942952 4549 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6966161f41a458e91ad757e8ac08b49 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f6966161f41a458e91ad757e8ac08b49. Latest consensus state: current_term: 1 leader_uuid: "f6966161f41a458e91ad757e8ac08b49" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6966161f41a458e91ad757e8ac08b49" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45271 } } }
I20260929 17:53:21.943068 4549 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6966161f41a458e91ad757e8ac08b49 [sys.catalog]: This master's current role is: LEADER
I20260929 17:53:21.942935 4548 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6966161f41a458e91ad757e8ac08b49 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f6966161f41a458e91ad757e8ac08b49" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6966161f41a458e91ad757e8ac08b49" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45271 } } }
I20260929 17:53:21.943163 4548 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6966161f41a458e91ad757e8ac08b49 [sys.catalog]: This master's current role is: LEADER
W20260929 17:53:21.946784 4562 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f6966161f41a458e91ad757e8ac08b49: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 17:53:21.946861 4562 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 17:53:21.946915 4554 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 17:53:21.947608 4554 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 17:53:21.952829 4554 catalog_manager.cc:1347] Generated new cluster ID: 7f9d088e90904f22884142974634c0a5
I20260929 17:53:21.952906 4554 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 17:53:21.962565 4554 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 17:53:21.963490 4554 catalog_manager.cc:1504] Loading token signing keys...
I20260929 17:53:21.971598 4554 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f6966161f41a458e91ad757e8ac08b49: Generated new TSK 0
I20260929 17:53:21.972429 4554 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704375975685-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704375975685-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260929 17:53:22.004388 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskup3Soe/build/debug/bin/kudu
/tmp/dist-test-taskup3Soe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704375975685-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704375975685-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704375975685-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704375975685-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45271
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704375975685-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704375975685-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704375975685-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704375975685-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:53:22.112270 4568 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:53:22.112547 4568 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:53:22.112612 4568 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:53:22.116706 4568 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:53:22.116773 4568 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:53:22.116863 4568 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 17:53:22.122121 4568 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704375975685-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704375975685-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704375975685-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704375975685-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45271
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4568
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704375975685-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:53:22.123152 4568 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:53:22.124167 4568 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:53:22.133203 4574 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:53:22.133244 4573 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:53:22.133216 4576 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:53:22.133539 4568 server_base.cc:1034] running on GCE node
I20260929 17:53:22.133901 4568 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:53:22.134478 4568 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:53:22.135623 4568 hybrid_clock.cc:648] HybridClock initialized: now 1790704402135606 us; error 31 us; skew 500 ppm
Sep 29 17:53:22 dist-test-slave-53nh krb5kdc[4462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704402, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:53:22.138185 4568 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260929 17:53:22.138885 4568 fs_manager.cc:359] Metadata directory not provided
I20260929 17:53:22.138948 4568 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:53:22.139124 4568 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:53:22.141088 4568 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704375975685-392-0/minicluster-data/ts-0/data/instance:
uuid: "aa267a00eaed4a448e0dadf7edbc044a"
format_stamp: "Formatted at 2026-09-29 17:53:22 on dist-test-slave-53nh"
server_key: "938b3285e19886940880945a1a6be618"
server_key_iv: "65df90a51a5216fa845fbbe4e9afe0ba"
server_key_version: "encryptionkey@0"
I20260929 17:53:22.141607 4568 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704375975685-392-0/minicluster-data/ts-0/wal/instance:
uuid: "aa267a00eaed4a448e0dadf7edbc044a"
format_stamp: "Formatted at 2026-09-29 17:53:22 on dist-test-slave-53nh"
server_key: "938b3285e19886940880945a1a6be618"
server_key_iv: "65df90a51a5216fa845fbbe4e9afe0ba"
server_key_version: "encryptionkey@0"
I20260929 17:53:22.145323 4568 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 17:53:22.147660 4582 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:53:22.148869 4568 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 17:53:22.148972 4568 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704375975685-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704375975685-392-0/minicluster-data/ts-0/wal
uuid: "aa267a00eaed4a448e0dadf7edbc044a"
format_stamp: "Formatted at 2026-09-29 17:53:22 on dist-test-slave-53nh"
server_key: "938b3285e19886940880945a1a6be618"
server_key_iv: "65df90a51a5216fa845fbbe4e9afe0ba"
server_key_version: "encryptionkey@0"
I20260929 17:53:22.149113 4568 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704375975685-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704375975685-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704375975685-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:53:22.175603 4568 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:53:22.178680 4568 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:53:22.178854 4568 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:53:22.179414 4568 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 17:53:22.180227 4568 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:53:22.180274 4568 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:53:22.180325 4568 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:53:22.180352 4568 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 17:53:22.188403 4568 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33405
I20260929 17:53:22.188424 4695 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33405 every 8 connection(s)
I20260929 17:53:22.188927 4568 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704375975685-392-0/minicluster-data/ts-0/data/info.pb
I20260929 17:53:22.190667 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskup3Soe/build/debug/bin/kudu as pid 4568
I20260929 17:53:22.190778 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704375975685-392-0/minicluster-data/ts-0/wal/instance
I20260929 17:53:22.191092 392 external_mini_cluster.cc:1428] Setting key b9a118afcbb2acbe22aabe703041cc32
Sep 29 17:53:22 dist-test-slave-53nh krb5kdc[4462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704402, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260929 17:53:22.203069 4545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:22.190500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37137 (local address 127.0.98.62:45271)
0929 17:53:22.190616 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:22.190620 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:22.191619 (+ 999us) server_negotiation.cc:407] Connection header received
0929 17:53:22.193152 (+ 1533us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:22.193155 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:22.193206 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:22.193275 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:53:22.194378 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:22.194902 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:22.196428 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:22.196652 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:22.196734 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:22.199462 (+ 2728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:53:22.199465 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:53:22.199467 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:53:22.199491 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:53:22.200978 (+ 1487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:53:22.201606 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:53:22.201609 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:53:22.201610 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:53:22.201685 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:53:22.201979 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:53:22.201981 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:53:22.201983 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:53:22.202146 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:53:22.202224 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0929 17:53:22.202811 (+ 587us) server_negotiation.cc:299] Negotiation successful
0929 17:53:22.202914 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260929 17:53:22.203948 4698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:22.190985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45271 (local address 127.0.98.1:37137)
0929 17:53:22.191464 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:22.191517 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0929 17:53:22.192929 (+ 1412us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:53:22.193441 (+ 512us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:53:22.193448 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:53:22.193784 (+ 336us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:53:22.194249 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:22.194262 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:22.195020 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:22.195024 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:22.196300 (+ 1276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:22.196312 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:22.196796 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:22.196800 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:22.196995 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:22.197623 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:53:22.197644 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:53:22.199290 (+ 1646us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:53:22.201149 (+ 1859us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:53:22.201154 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:53:22.201166 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:53:22.201497 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:53:22.201762 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:53:22.201764 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:53:22.201765 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:53:22.201906 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:53:22.202250 (+ 344us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:53:22.202255 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:53:22.202586 (+ 331us) client_negotiation.cc:769] Sending connection context
0929 17:53:22.202754 (+ 168us) client_negotiation.cc:240] Negotiation successful
0929 17:53:22.203079 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":135,"threads_started":1}
I20260929 17:53:22.205179 4696 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45271
I20260929 17:53:22.205401 4696 heartbeater.cc:464] Registering TS with master...
I20260929 17:53:22.205832 4696 heartbeater.cc:511] Master 127.0.98.62:45271 requested a full tablet report, sending...
I20260929 17:53:22.207257 4506 ts_manager.cc:194] Registered new tserver with Master: aa267a00eaed4a448e0dadf7edbc044a (127.0.98.1:33405)
I20260929 17:53:22.208734 4506 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:37137
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704375975685-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704375975685-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260929 17:53:22.217734 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskup3Soe/build/debug/bin/kudu
/tmp/dist-test-taskup3Soe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704375975685-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704375975685-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704375975685-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704375975685-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45271
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704375975685-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704375975685-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704375975685-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704375975685-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:53:22.324321 4701 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:53:22.324589 4701 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:53:22.324632 4701 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:53:22.328634 4701 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:53:22.328703 4701 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:53:22.328768 4701 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 17:53:22.334218 4701 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704375975685-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704375975685-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704375975685-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704375975685-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45271
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4701
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704375975685-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:53:22.335280 4701 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:53:22.336269 4701 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 17:53:22.345613 4701 server_base.cc:1034] running on GCE node
W20260929 17:53:22.345568 4706 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:53:22.345597 4707 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:53:22.345568 4709 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:53:22.346258 4701 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:53:22.346853 4701 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:53:22.348001 4701 hybrid_clock.cc:648] HybridClock initialized: now 1790704402347957 us; error 55 us; skew 500 ppm
Sep 29 17:53:22 dist-test-slave-53nh krb5kdc[4462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704402, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:53:22.350641 4701 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260929 17:53:22.351361 4701 fs_manager.cc:359] Metadata directory not provided
I20260929 17:53:22.351428 4701 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:53:22.351608 4701 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:53:22.353719 4701 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704375975685-392-0/minicluster-data/ts-1/data/instance:
uuid: "183882187043422882fa96e52130ed82"
format_stamp: "Formatted at 2026-09-29 17:53:22 on dist-test-slave-53nh"
server_key: "0b3b8b9a8c491d38b45eb84b74c6046e"
server_key_iv: "f5d5b3fbe49f72d61b2e9c9ff843566d"
server_key_version: "encryptionkey@0"
I20260929 17:53:22.354242 4701 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704375975685-392-0/minicluster-data/ts-1/wal/instance:
uuid: "183882187043422882fa96e52130ed82"
format_stamp: "Formatted at 2026-09-29 17:53:22 on dist-test-slave-53nh"
server_key: "0b3b8b9a8c491d38b45eb84b74c6046e"
server_key_iv: "f5d5b3fbe49f72d61b2e9c9ff843566d"
server_key_version: "encryptionkey@0"
I20260929 17:53:22.357784 4701 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260929 17:53:22.360064 4715 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:53:22.361500 4701 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 17:53:22.361613 4701 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704375975685-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704375975685-392-0/minicluster-data/ts-1/wal
uuid: "183882187043422882fa96e52130ed82"
format_stamp: "Formatted at 2026-09-29 17:53:22 on dist-test-slave-53nh"
server_key: "0b3b8b9a8c491d38b45eb84b74c6046e"
server_key_iv: "f5d5b3fbe49f72d61b2e9c9ff843566d"
server_key_version: "encryptionkey@0"
I20260929 17:53:22.361707 4701 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704375975685-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704375975685-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704375975685-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:53:22.382990 4701 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:53:22.385713 4701 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:53:22.385881 4701 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:53:22.386430 4701 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 17:53:22.387205 4701 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:53:22.387269 4701 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:53:22.387319 4701 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:53:22.387347 4701 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 17:53:22.395474 4701 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41173
I20260929 17:53:22.395493 4828 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41173 every 8 connection(s)
I20260929 17:53:22.396031 4701 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704375975685-392-0/minicluster-data/ts-1/data/info.pb
I20260929 17:53:22.403246 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskup3Soe/build/debug/bin/kudu as pid 4701
I20260929 17:53:22.403404 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704375975685-392-0/minicluster-data/ts-1/wal/instance
I20260929 17:53:22.403715 392 external_mini_cluster.cc:1428] Setting key 2111a1b0a66337129e7492615eec2e44
Sep 29 17:53:22 dist-test-slave-53nh krb5kdc[4462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704402, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 17:53:22.409278 4545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:22.397636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58885 (local address 127.0.98.62:45271)
0929 17:53:22.397794 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:22.397799 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:22.398616 (+ 817us) server_negotiation.cc:407] Connection header received
0929 17:53:22.399914 (+ 1298us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:22.399917 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:22.399962 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:22.400023 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:53:22.401283 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:22.401805 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:22.402876 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:22.403093 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:22.403171 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:22.405831 (+ 2660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:53:22.405835 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:53:22.405838 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:53:22.405865 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:53:22.407203 (+ 1338us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:53:22.407751 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:53:22.407755 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:53:22.407756 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:53:22.407854 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:53:22.408140 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:53:22.408143 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:53:22.408145 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:53:22.408352 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:53:22.408437 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0929 17:53:22.408942 (+ 505us) server_negotiation.cc:299] Negotiation successful
0929 17:53:22.409110 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 17:53:22.410065 4831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:22.398037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45271 (local address 127.0.98.2:58885)
0929 17:53:22.398468 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:22.398511 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 17:53:22.399697 (+ 1186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:53:22.400218 (+ 521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:53:22.400227 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:53:22.400588 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:53:22.401133 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:22.401147 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:22.401906 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:22.401908 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:22.402769 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:22.402778 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:22.403251 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:22.403256 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:22.403455 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:22.404061 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:53:22.404082 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:53:22.405675 (+ 1593us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:53:22.407311 (+ 1636us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:53:22.407317 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:53:22.407332 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:53:22.407660 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:53:22.407938 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:53:22.407940 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:53:22.407941 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:53:22.408069 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:53:22.408439 (+ 370us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:53:22.408444 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:53:22.408673 (+ 229us) client_negotiation.cc:769] Sending connection context
0929 17:53:22.408880 (+ 207us) client_negotiation.cc:240] Negotiation successful
0929 17:53:22.409206 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":106,"threads_started":1}
I20260929 17:53:22.411125 4829 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45271
I20260929 17:53:22.411365 4829 heartbeater.cc:464] Registering TS with master...
I20260929 17:53:22.411787 4829 heartbeater.cc:511] Master 127.0.98.62:45271 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260929 17:53:22.412772 4506 ts_manager.cc:194] Registered new tserver with Master: 183882187043422882fa96e52130ed82 (127.0.98.2:41173)
I20260929 17:53:22.413354 4506 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:58885
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704375975685-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704375975685-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260929 17:53:22.428658 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskup3Soe/build/debug/bin/kudu
/tmp/dist-test-taskup3Soe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704375975685-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704375975685-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704375975685-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704375975685-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45271
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704375975685-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704375975685-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704375975685-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704375975685-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:53:22.536656 4834 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:53:22.536906 4834 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:53:22.536963 4834 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:53:22.540876 4834 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:53:22.540934 4834 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:53:22.541033 4834 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 17:53:22.546422 4834 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704375975685-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704375975685-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704375975685-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704375975685-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45271
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4834
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704375975685-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:53:22.547428 4834 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:53:22.548420 4834 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:53:22.557421 4840 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:53:22.557530 4839 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:53:22.557421 4842 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:53:22.557770 4834 server_base.cc:1034] running on GCE node
I20260929 17:53:22.558197 4834 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:53:22.558833 4834 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:53:22.559986 4834 hybrid_clock.cc:648] HybridClock initialized: now 1790704402559962 us; error 35 us; skew 500 ppm
Sep 29 17:53:22 dist-test-slave-53nh krb5kdc[4462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704402, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:53:22.562610 4834 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260929 17:53:22.563333 4834 fs_manager.cc:359] Metadata directory not provided
I20260929 17:53:22.563403 4834 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:53:22.563587 4834 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:53:22.565768 4834 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704375975685-392-0/minicluster-data/ts-2/data/instance:
uuid: "ee58869ee79d4beba9a998d90ac93364"
format_stamp: "Formatted at 2026-09-29 17:53:22 on dist-test-slave-53nh"
server_key: "17c4207f0f20a46ab3cd1b8902cd05a7"
server_key_iv: "cfe7c587aa801cae8a2da782cf94d25f"
server_key_version: "encryptionkey@0"
I20260929 17:53:22.566283 4834 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704375975685-392-0/minicluster-data/ts-2/wal/instance:
uuid: "ee58869ee79d4beba9a998d90ac93364"
format_stamp: "Formatted at 2026-09-29 17:53:22 on dist-test-slave-53nh"
server_key: "17c4207f0f20a46ab3cd1b8902cd05a7"
server_key_iv: "cfe7c587aa801cae8a2da782cf94d25f"
server_key_version: "encryptionkey@0"
I20260929 17:53:22.569851 4834 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260929 17:53:22.572150 4848 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:53:22.573403 4834 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260929 17:53:22.573503 4834 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704375975685-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704375975685-392-0/minicluster-data/ts-2/wal
uuid: "ee58869ee79d4beba9a998d90ac93364"
format_stamp: "Formatted at 2026-09-29 17:53:22 on dist-test-slave-53nh"
server_key: "17c4207f0f20a46ab3cd1b8902cd05a7"
server_key_iv: "cfe7c587aa801cae8a2da782cf94d25f"
server_key_version: "encryptionkey@0"
I20260929 17:53:22.573606 4834 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704375975685-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704375975685-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704375975685-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:53:22.586815 4834 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:53:22.589787 4834 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:53:22.589947 4834 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:53:22.590461 4834 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 17:53:22.591245 4834 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:53:22.591290 4834 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:53:22.591327 4834 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:53:22.591346 4834 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 17:53:22.599436 4834 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34105
I20260929 17:53:22.599519 4961 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34105 every 8 connection(s)
I20260929 17:53:22.600039 4834 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704375975685-392-0/minicluster-data/ts-2/data/info.pb
I20260929 17:53:22.604362 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskup3Soe/build/debug/bin/kudu as pid 4834
I20260929 17:53:22.604478 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704375975685-392-0/minicluster-data/ts-2/wal/instance
I20260929 17:53:22.604738 392 external_mini_cluster.cc:1428] Setting key 3dee0a55250a8e4099e731a328e72f8d
Sep 29 17:53:22 dist-test-slave-53nh krb5kdc[4462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704402, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 17:53:22.612864 4545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:22.601941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60867 (local address 127.0.98.62:45271)
0929 17:53:22.602094 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:22.602100 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:22.602942 (+ 842us) server_negotiation.cc:407] Connection header received
0929 17:53:22.604287 (+ 1345us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:22.604291 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:22.604347 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:22.604441 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:53:22.605725 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:22.606322 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:22.607471 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:22.607678 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:22.607754 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:22.610084 (+ 2330us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:53:22.610087 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:53:22.610089 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:53:22.610111 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:53:22.611245 (+ 1134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:53:22.611675 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:53:22.611678 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:53:22.611679 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:53:22.611742 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:53:22.611979 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:53:22.611981 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:53:22.611982 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:53:22.612132 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:53:22.612201 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0929 17:53:22.612643 (+ 442us) server_negotiation.cc:299] Negotiation successful
0929 17:53:22.612732 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 17:53:22.613674 4964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:22.602315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45271 (local address 127.0.98.3:60867)
0929 17:53:22.602781 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:22.602826 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 17:53:22.604083 (+ 1257us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:53:22.604645 (+ 562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:53:22.604653 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:53:22.605100 (+ 447us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:53:22.605587 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:22.605605 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:22.606445 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:22.606450 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:22.607366 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:22.607376 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:22.607806 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:22.607810 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:22.607954 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:22.608482 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:53:22.608499 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:53:22.609918 (+ 1419us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:53:22.611340 (+ 1422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:53:22.611345 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:53:22.611356 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:53:22.611605 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:53:22.611798 (+ 193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:53:22.611800 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:53:22.611801 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:53:22.611919 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:53:22.612235 (+ 316us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:53:22.612240 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:53:22.612462 (+ 222us) client_negotiation.cc:769] Sending connection context
0929 17:53:22.612617 (+ 155us) client_negotiation.cc:240] Negotiation successful
0929 17:53:22.612872 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":112,"threads_started":1}
I20260929 17:53:22.614730 4962 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45271
I20260929 17:53:22.614943 4962 heartbeater.cc:464] Registering TS with master...
I20260929 17:53:22.615370 4962 heartbeater.cc:511] Master 127.0.98.62:45271 requested a full tablet report, sending...
I20260929 17:53:22.616298 4506 ts_manager.cc:194] Registered new tserver with Master: ee58869ee79d4beba9a998d90ac93364 (127.0.98.3:34105)
I20260929 17:53:22.616818 4506 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:60867
I20260929 17:53:22.618320 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 17:53:22.626220 4545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:22.620030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53604 (local address 127.0.98.62:45271)
0929 17:53:22.620127 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:22.620130 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:22.620304 (+ 174us) server_negotiation.cc:407] Connection header received
0929 17:53:22.620393 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:22.620396 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:22.620427 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:22.620486 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:53:22.621081 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:22.621570 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:22.622365 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:22.622558 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:22.622622 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:22.623719 (+ 1097us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:53:22.623721 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:53:22.623724 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:53:22.623741 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:53:22.624940 (+ 1199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:53:22.625316 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:53:22.625319 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:53:22.625320 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:53:22.625391 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:53:22.625580 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:53:22.625582 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:53:22.625583 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:53:22.625781 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:53:22.625872 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0929 17:53:22.626031 (+ 159us) server_negotiation.cc:299] Negotiation successful
0929 17:53:22.626112 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260929 17:53:22.628731 4506 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53604:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 17:53:22.631610 4506 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 17:53:22.645717 4974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:22.639609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38186 (local address 127.0.98.3:34105)
0929 17:53:22.640654 (+ 1045us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:22.640667 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:22.640692 (+ 25us) server_negotiation.cc:407] Connection header received
0929 17:53:22.640767 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:22.640773 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:22.640932 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:22.641085 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:53:22.641539 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:22.642494 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:22.644381 (+ 1887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:22.645068 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:22.645168 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:22.645331 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0929 17:53:22.645464 (+ 133us) server_negotiation.cc:299] Negotiation successful
0929 17:53:22.645562 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":857,"thread_start_us":102,"threads_started":1}
I20260929 17:53:22.645712 4973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:22.639476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34105 (local address 127.0.0.1:38186)
0929 17:53:22.639771 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:22.639819 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0929 17:53:22.640151 (+ 332us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:53:22.641189 (+ 1038us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:53:22.641195 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:53:22.641214 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:53:22.641416 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:22.641424 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:22.642609 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:22.642613 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:22.644221 (+ 1608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:22.644232 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:22.645172 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:22.645175 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:22.645280 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:22.645332 (+ 52us) client_negotiation.cc:769] Sending connection context
0929 17:53:22.645438 (+ 106us) client_negotiation.cc:240] Negotiation successful
0929 17:53:22.645570 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"mutex_wait_us":86,"thread_start_us":61,"threads_started":1}
I20260929 17:53:22.645905 4975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:22.639720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33405 (local address 127.0.0.1:36434)
0929 17:53:22.640651 (+ 931us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:22.640664 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 17:53:22.640764 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:53:22.641279 (+ 515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:53:22.641282 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:53:22.641298 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:53:22.641452 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:22.641459 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:22.642887 (+ 1428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:22.642890 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:22.644221 (+ 1331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:22.644232 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:22.645418 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:22.645422 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:22.645578 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:22.645613 (+ 35us) client_negotiation.cc:769] Sending connection context
0929 17:53:22.645669 (+ 56us) client_negotiation.cc:240] Negotiation successful
0929 17:53:22.645764 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":876,"mutex_wait_us":206,"thread_start_us":96,"threads_started":1}
I20260929 17:53:22.646237 4976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:22.640126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41173 (local address 127.0.0.1:51212)
0929 17:53:22.640938 (+ 812us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:22.640949 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 17:53:22.641061 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:53:22.641483 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:53:22.641486 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:53:22.641503 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:53:22.641606 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:22.641611 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:22.643203 (+ 1592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:22.643206 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:22.644120 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:22.644128 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:22.645418 (+ 1290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:22.645422 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:22.645554 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:22.645603 (+ 49us) client_negotiation.cc:769] Sending connection context
0929 17:53:22.645669 (+ 66us) client_negotiation.cc:240] Negotiation successful
0929 17:53:22.646127 (+ 458us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":759,"mutex_wait_us":412,"thread_start_us":108,"threads_started":1}
I20260929 17:53:22.646524 4978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:22.640282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51212 (local address 127.0.98.2:41173)
0929 17:53:22.640655 (+ 373us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:22.640666 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:22.641056 (+ 390us) server_negotiation.cc:407] Connection header received
0929 17:53:22.641159 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:22.641164 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:22.641328 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:22.641490 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:53:22.641704 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:22.643075 (+ 1371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:22.644255 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:22.645110 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:22.645223 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:22.645358 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0929 17:53:22.645749 (+ 391us) server_negotiation.cc:299] Negotiation successful
0929 17:53:22.646270 (+ 521us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":296,"thread_start_us":132,"threads_started":1}
I20260929 17:53:22.646623 4977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:22.640104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36434 (local address 127.0.98.1:33405)
0929 17:53:22.640526 (+ 422us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:22.640538 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:22.640742 (+ 204us) server_negotiation.cc:407] Connection header received
0929 17:53:22.640874 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:22.640880 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:22.641106 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:22.641242 (+ 136us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:53:22.641578 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:22.642760 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:22.644337 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:22.645135 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:22.645221 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:22.645358 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0929 17:53:22.646392 (+ 1034us) server_negotiation.cc:299] Negotiation successful
0929 17:53:22.646490 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":332,"thread_start_us":110,"threads_started":1}
I20260929 17:53:22.648921 4630 tablet_service.cc:1467] Processing CreateTablet for tablet 60a86e20980841d29150e56819d942bd (DEFAULT_TABLE table=test-table [id=0f30ae76db8c494ea59f60bf1ac95c8e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 17:53:22.648948 4896 tablet_service.cc:1467] Processing CreateTablet for tablet 60a86e20980841d29150e56819d942bd (DEFAULT_TABLE table=test-table [id=0f30ae76db8c494ea59f60bf1ac95c8e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 17:53:22.649253 4763 tablet_service.cc:1467] Processing CreateTablet for tablet 60a86e20980841d29150e56819d942bd (DEFAULT_TABLE table=test-table [id=0f30ae76db8c494ea59f60bf1ac95c8e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 17:53:22.649960 4630 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60a86e20980841d29150e56819d942bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:53:22.650010 4763 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60a86e20980841d29150e56819d942bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:53:22.650022 4896 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60a86e20980841d29150e56819d942bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:53:22.656045 4979 tablet_bootstrap.cc:492] T 60a86e20980841d29150e56819d942bd P aa267a00eaed4a448e0dadf7edbc044a: Bootstrap starting.
I20260929 17:53:22.657555 4981 tablet_bootstrap.cc:492] T 60a86e20980841d29150e56819d942bd P ee58869ee79d4beba9a998d90ac93364: Bootstrap starting.
I20260929 17:53:22.658743 4979 tablet_bootstrap.cc:654] T 60a86e20980841d29150e56819d942bd P aa267a00eaed4a448e0dadf7edbc044a: Neither blocks nor log segments found. Creating new log.
I20260929 17:53:22.659205 4981 tablet_bootstrap.cc:654] T 60a86e20980841d29150e56819d942bd P ee58869ee79d4beba9a998d90ac93364: Neither blocks nor log segments found. Creating new log.
I20260929 17:53:22.659786 4979 log.cc:824] T 60a86e20980841d29150e56819d942bd P aa267a00eaed4a448e0dadf7edbc044a: Log is configured to *not* fsync() on all Append() calls
I20260929 17:53:22.660138 4981 log.cc:824] T 60a86e20980841d29150e56819d942bd P ee58869ee79d4beba9a998d90ac93364: Log is configured to *not* fsync() on all Append() calls
I20260929 17:53:22.660444 4980 tablet_bootstrap.cc:492] T 60a86e20980841d29150e56819d942bd P 183882187043422882fa96e52130ed82: Bootstrap starting.
I20260929 17:53:22.661842 4981 tablet_bootstrap.cc:492] T 60a86e20980841d29150e56819d942bd P ee58869ee79d4beba9a998d90ac93364: No bootstrap required, opened a new log
I20260929 17:53:22.662014 4981 ts_tablet_manager.cc:1397] T 60a86e20980841d29150e56819d942bd P ee58869ee79d4beba9a998d90ac93364: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260929 17:53:22.662283 4979 tablet_bootstrap.cc:492] T 60a86e20980841d29150e56819d942bd P aa267a00eaed4a448e0dadf7edbc044a: No bootstrap required, opened a new log
I20260929 17:53:22.662451 4980 tablet_bootstrap.cc:654] T 60a86e20980841d29150e56819d942bd P 183882187043422882fa96e52130ed82: Neither blocks nor log segments found. Creating new log.
I20260929 17:53:22.662484 4979 ts_tablet_manager.cc:1397] T 60a86e20980841d29150e56819d942bd P aa267a00eaed4a448e0dadf7edbc044a: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260929 17:53:22.663565 4980 log.cc:824] T 60a86e20980841d29150e56819d942bd P 183882187043422882fa96e52130ed82: Log is configured to *not* fsync() on all Append() calls
I20260929 17:53:22.665405 4981 raft_consensus.cc:348] T 60a86e20980841d29150e56819d942bd P ee58869ee79d4beba9a998d90ac93364 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee58869ee79d4beba9a998d90ac93364" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34105 } } peers { permanent_uuid: "183882187043422882fa96e52130ed82" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41173 } } peers { permanent_uuid: "aa267a00eaed4a448e0dadf7edbc044a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33405 } }
I20260929 17:53:22.665616 4980 tablet_bootstrap.cc:492] T 60a86e20980841d29150e56819d942bd P 183882187043422882fa96e52130ed82: No bootstrap required, opened a new log
I20260929 17:53:22.665685 4981 raft_consensus.cc:374] T 60a86e20980841d29150e56819d942bd P ee58869ee79d4beba9a998d90ac93364 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:53:22.665742 4981 raft_consensus.cc:729] T 60a86e20980841d29150e56819d942bd P ee58869ee79d4beba9a998d90ac93364 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee58869ee79d4beba9a998d90ac93364, State: Initialized, Role: FOLLOWER
I20260929 17:53:22.665792 4980 ts_tablet_manager.cc:1397] T 60a86e20980841d29150e56819d942bd P 183882187043422882fa96e52130ed82: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 17:53:22.666158 4981 consensus_queue.cc:260] T 60a86e20980841d29150e56819d942bd P ee58869ee79d4beba9a998d90ac93364 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee58869ee79d4beba9a998d90ac93364" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34105 } } peers { permanent_uuid: "183882187043422882fa96e52130ed82" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41173 } } peers { permanent_uuid: "aa267a00eaed4a448e0dadf7edbc044a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33405 } }
I20260929 17:53:22.666095 4979 raft_consensus.cc:348] T 60a86e20980841d29150e56819d942bd P aa267a00eaed4a448e0dadf7edbc044a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee58869ee79d4beba9a998d90ac93364" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34105 } } peers { permanent_uuid: "183882187043422882fa96e52130ed82" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41173 } } peers { permanent_uuid: "aa267a00eaed4a448e0dadf7edbc044a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33405 } }
I20260929 17:53:22.666356 4979 raft_consensus.cc:374] T 60a86e20980841d29150e56819d942bd P aa267a00eaed4a448e0dadf7edbc044a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:53:22.666409 4979 raft_consensus.cc:729] T 60a86e20980841d29150e56819d942bd P aa267a00eaed4a448e0dadf7edbc044a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa267a00eaed4a448e0dadf7edbc044a, State: Initialized, Role: FOLLOWER
I20260929 17:53:22.666836 4979 consensus_queue.cc:260] T 60a86e20980841d29150e56819d942bd P aa267a00eaed4a448e0dadf7edbc044a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee58869ee79d4beba9a998d90ac93364" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34105 } } peers { permanent_uuid: "183882187043422882fa96e52130ed82" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41173 } } peers { permanent_uuid: "aa267a00eaed4a448e0dadf7edbc044a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33405 } }
I20260929 17:53:22.667038 4981 ts_tablet_manager.cc:1428] T 60a86e20980841d29150e56819d942bd P ee58869ee79d4beba9a998d90ac93364: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260929 17:53:22.667378 4962 heartbeater.cc:503] Master 127.0.98.62:45271 was elected leader, sending a full tablet report...
I20260929 17:53:22.667776 4979 ts_tablet_manager.cc:1428] T 60a86e20980841d29150e56819d942bd P aa267a00eaed4a448e0dadf7edbc044a: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260929 17:53:22.668109 4696 heartbeater.cc:503] Master 127.0.98.62:45271 was elected leader, sending a full tablet report...
I20260929 17:53:22.669302 4980 raft_consensus.cc:348] T 60a86e20980841d29150e56819d942bd P 183882187043422882fa96e52130ed82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee58869ee79d4beba9a998d90ac93364" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34105 } } peers { permanent_uuid: "183882187043422882fa96e52130ed82" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41173 } } peers { permanent_uuid: "aa267a00eaed4a448e0dadf7edbc044a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33405 } }
I20260929 17:53:22.669569 4980 raft_consensus.cc:374] T 60a86e20980841d29150e56819d942bd P 183882187043422882fa96e52130ed82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:53:22.669631 4980 raft_consensus.cc:729] T 60a86e20980841d29150e56819d942bd P 183882187043422882fa96e52130ed82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 183882187043422882fa96e52130ed82, State: Initialized, Role: FOLLOWER
I20260929 17:53:22.670033 4980 consensus_queue.cc:260] T 60a86e20980841d29150e56819d942bd P 183882187043422882fa96e52130ed82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee58869ee79d4beba9a998d90ac93364" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34105 } } peers { permanent_uuid: "183882187043422882fa96e52130ed82" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41173 } } peers { permanent_uuid: "aa267a00eaed4a448e0dadf7edbc044a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33405 } }
I20260929 17:53:22.670723 4829 heartbeater.cc:503] Master 127.0.98.62:45271 was elected leader, sending a full tablet report...
I20260929 17:53:22.670920 4980 ts_tablet_manager.cc:1428] T 60a86e20980841d29150e56819d942bd P 183882187043422882fa96e52130ed82: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
W20260929 17:53:22.690081 4697 tablet.cc:2369] T 60a86e20980841d29150e56819d942bd P aa267a00eaed4a448e0dadf7edbc044a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 17:53:22.789439 4986 raft_consensus.cc:482] T 60a86e20980841d29150e56819d942bd P aa267a00eaed4a448e0dadf7edbc044a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 17:53:22.789656 4986 raft_consensus.cc:504] T 60a86e20980841d29150e56819d942bd P aa267a00eaed4a448e0dadf7edbc044a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee58869ee79d4beba9a998d90ac93364" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34105 } } peers { permanent_uuid: "183882187043422882fa96e52130ed82" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41173 } } peers { permanent_uuid: "aa267a00eaed4a448e0dadf7edbc044a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33405 } }
I20260929 17:53:22.790755 4986 leader_election.cc:288] T 60a86e20980841d29150e56819d942bd P aa267a00eaed4a448e0dadf7edbc044a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ee58869ee79d4beba9a998d90ac93364 (127.0.98.3:34105), 183882187043422882fa96e52130ed82 (127.0.98.2:41173)
I20260929 17:53:22.794585 4978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:22.791476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55959 (local address 127.0.98.2:41173)
0929 17:53:22.791595 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:22.791599 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:22.791631 (+ 32us) server_negotiation.cc:407] Connection header received
0929 17:53:22.791764 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:22.791766 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:22.791813 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:22.791884 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:53:22.792126 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:22.792634 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:22.793673 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:22.794051 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:22.794134 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:22.794193 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0929 17:53:22.794398 (+ 205us) server_negotiation.cc:299] Negotiation successful
0929 17:53:22.794459 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 17:53:22.794598 4974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:22.791079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40497 (local address 127.0.98.3:34105)
0929 17:53:22.791205 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:22.791210 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:22.791322 (+ 112us) server_negotiation.cc:407] Connection header received
0929 17:53:22.791452 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:22.791456 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:22.791506 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:22.791572 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:53:22.791883 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:22.792484 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:22.793646 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:22.794056 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:22.794148 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:22.794205 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0929 17:53:22.794419 (+ 214us) server_negotiation.cc:299] Negotiation successful
0929 17:53:22.794480 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 17:53:22.794616 4989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:22.791322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41173 (local address 127.0.98.1:55959)
0929 17:53:22.791574 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:22.791585 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 17:53:22.791671 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:53:22.791897 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:53:22.791899 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:53:22.791915 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:53:22.792028 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:22.792035 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:22.792731 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:22.792734 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:22.793558 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:22.793567 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:22.794159 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:22.794162 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:22.794253 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:22.794309 (+ 56us) client_negotiation.cc:769] Sending connection context
0929 17:53:22.794364 (+ 55us) client_negotiation.cc:240] Negotiation successful
0929 17:53:22.794433 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":193,"mutex_wait_us":38,"thread_start_us":112,"threads_started":1}
I20260929 17:53:22.794631 4988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:22.790957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34105 (local address 127.0.98.1:40497)
0929 17:53:22.791225 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:22.791239 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 17:53:22.791352 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:53:22.791597 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:53:22.791600 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:53:22.791624 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:53:22.791771 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:22.791779 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:22.792588 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:22.792592 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:22.793529 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:22.793538 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:22.794170 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:22.794173 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:22.794289 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:22.794325 (+ 36us) client_negotiation.cc:769] Sending connection context
0929 17:53:22.794391 (+ 66us) client_negotiation.cc:240] Negotiation successful
0929 17:53:22.794459 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202,"mutex_wait_us":18,"spinlock_wait_cycles":6016,"thread_start_us":103,"threads_started":1}
I20260929 17:53:22.795243 4916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60a86e20980841d29150e56819d942bd" candidate_uuid: "aa267a00eaed4a448e0dadf7edbc044a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee58869ee79d4beba9a998d90ac93364" is_pre_election: true
I20260929 17:53:22.795287 4783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60a86e20980841d29150e56819d942bd" candidate_uuid: "aa267a00eaed4a448e0dadf7edbc044a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "183882187043422882fa96e52130ed82" is_pre_election: true
I20260929 17:53:22.795558 4783 raft_consensus.cc:2454] T 60a86e20980841d29150e56819d942bd P 183882187043422882fa96e52130ed82 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aa267a00eaed4a448e0dadf7edbc044a in term 0.
I20260929 17:53:22.795559 4916 raft_consensus.cc:2454] T 60a86e20980841d29150e56819d942bd P ee58869ee79d4beba9a998d90ac93364 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aa267a00eaed4a448e0dadf7edbc044a in term 0.
I20260929 17:53:22.795989 4586 leader_election.cc:302] T 60a86e20980841d29150e56819d942bd P aa267a00eaed4a448e0dadf7edbc044a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aa267a00eaed4a448e0dadf7edbc044a, ee58869ee79d4beba9a998d90ac93364; no voters:
I20260929 17:53:22.796221 4986 raft_consensus.cc:2781] T 60a86e20980841d29150e56819d942bd P aa267a00eaed4a448e0dadf7edbc044a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 17:53:22.796310 4986 raft_consensus.cc:482] T 60a86e20980841d29150e56819d942bd P aa267a00eaed4a448e0dadf7edbc044a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 17:53:22.796360 4986 raft_consensus.cc:3037] T 60a86e20980841d29150e56819d942bd P aa267a00eaed4a448e0dadf7edbc044a [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:53:22.797353 4986 raft_consensus.cc:504] T 60a86e20980841d29150e56819d942bd P aa267a00eaed4a448e0dadf7edbc044a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee58869ee79d4beba9a998d90ac93364" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34105 } } peers { permanent_uuid: "183882187043422882fa96e52130ed82" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41173 } } peers { permanent_uuid: "aa267a00eaed4a448e0dadf7edbc044a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33405 } }
I20260929 17:53:22.797647 4986 leader_election.cc:288] T 60a86e20980841d29150e56819d942bd P aa267a00eaed4a448e0dadf7edbc044a [CANDIDATE]: Term 1 election: Requested vote from peers ee58869ee79d4beba9a998d90ac93364 (127.0.98.3:34105), 183882187043422882fa96e52130ed82 (127.0.98.2:41173)
I20260929 17:53:22.797936 4916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60a86e20980841d29150e56819d942bd" candidate_uuid: "aa267a00eaed4a448e0dadf7edbc044a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee58869ee79d4beba9a998d90ac93364"
I20260929 17:53:22.798008 4783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60a86e20980841d29150e56819d942bd" candidate_uuid: "aa267a00eaed4a448e0dadf7edbc044a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "183882187043422882fa96e52130ed82"
I20260929 17:53:22.798049 4916 raft_consensus.cc:3037] T 60a86e20980841d29150e56819d942bd P ee58869ee79d4beba9a998d90ac93364 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:53:22.798132 4783 raft_consensus.cc:3037] T 60a86e20980841d29150e56819d942bd P 183882187043422882fa96e52130ed82 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:53:22.799027 4916 raft_consensus.cc:2454] T 60a86e20980841d29150e56819d942bd P ee58869ee79d4beba9a998d90ac93364 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aa267a00eaed4a448e0dadf7edbc044a in term 1.
I20260929 17:53:22.799134 4783 raft_consensus.cc:2454] T 60a86e20980841d29150e56819d942bd P 183882187043422882fa96e52130ed82 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aa267a00eaed4a448e0dadf7edbc044a in term 1.
I20260929 17:53:22.799290 4586 leader_election.cc:302] T 60a86e20980841d29150e56819d942bd P aa267a00eaed4a448e0dadf7edbc044a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aa267a00eaed4a448e0dadf7edbc044a, ee58869ee79d4beba9a998d90ac93364; no voters:
I20260929 17:53:22.799453 4986 raft_consensus.cc:2781] T 60a86e20980841d29150e56819d942bd P aa267a00eaed4a448e0dadf7edbc044a [term 1 FOLLOWER]: Leader election won for term 1
I20260929 17:53:22.799674 4986 raft_consensus.cc:686] T 60a86e20980841d29150e56819d942bd P aa267a00eaed4a448e0dadf7edbc044a [term 1 LEADER]: Becoming Leader. State: Replica: aa267a00eaed4a448e0dadf7edbc044a, State: Running, Role: LEADER
I20260929 17:53:22.800001 4986 consensus_queue.cc:237] T 60a86e20980841d29150e56819d942bd P aa267a00eaed4a448e0dadf7edbc044a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee58869ee79d4beba9a998d90ac93364" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34105 } } peers { permanent_uuid: "183882187043422882fa96e52130ed82" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41173 } } peers { permanent_uuid: "aa267a00eaed4a448e0dadf7edbc044a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33405 } }
I20260929 17:53:22.803164 4505 catalog_manager.cc:5495] T 60a86e20980841d29150e56819d942bd P aa267a00eaed4a448e0dadf7edbc044a reported cstate change: term changed from 0 to 1, leader changed from <none> to aa267a00eaed4a448e0dadf7edbc044a (127.0.98.1). New cstate: current_term: 1 leader_uuid: "aa267a00eaed4a448e0dadf7edbc044a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee58869ee79d4beba9a998d90ac93364" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34105 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "183882187043422882fa96e52130ed82" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41173 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aa267a00eaed4a448e0dadf7edbc044a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33405 } health_report { overall_health: HEALTHY } } }
I20260929 17:53:22.833637 4977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:22.830572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36440 (local address 127.0.98.1:33405)
0929 17:53:22.830684 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:22.830688 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:22.830701 (+ 13us) server_negotiation.cc:407] Connection header received
0929 17:53:22.830776 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:22.830778 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:22.830838 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:22.830920 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 17:53:22.831104 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:22.831612 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:22.832380 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:22.832587 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:22.832655 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:22.832873 (+ 218us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 17:53:22.833356 (+ 483us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 17:53:22.833446 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0929 17:53:22.833477 (+ 31us) server_negotiation.cc:299] Negotiation successful
0929 17:53:22.833525 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260929 17:53:22.839442 4916 raft_consensus.cc:1261] T 60a86e20980841d29150e56819d942bd P ee58869ee79d4beba9a998d90ac93364 [term 1 FOLLOWER]: Refusing update from remote peer aa267a00eaed4a448e0dadf7edbc044a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 17:53:22.839442 4783 raft_consensus.cc:1261] T 60a86e20980841d29150e56819d942bd P 183882187043422882fa96e52130ed82 [term 1 FOLLOWER]: Refusing update from remote peer aa267a00eaed4a448e0dadf7edbc044a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 17:53:22.840135 4990 consensus_queue.cc:1035] T 60a86e20980841d29150e56819d942bd P aa267a00eaed4a448e0dadf7edbc044a [LEADER]: Connected to new peer: Peer: permanent_uuid: "183882187043422882fa96e52130ed82" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 17:53:22.840312 4986 consensus_queue.cc:1035] T 60a86e20980841d29150e56819d942bd P aa267a00eaed4a448e0dadf7edbc044a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee58869ee79d4beba9a998d90ac93364" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 17:53:22.847209 4993 mvcc.cc:205] Tried to move back new op lower bound from 7334725234022993920 to 7334725233871536128. Current Snapshot: MvccSnapshot[applied={T|T < 7334725234022993920}]
I20260929 17:53:22.847910 4995 mvcc.cc:205] Tried to move back new op lower bound from 7334725234022993920 to 7334725233871536128. Current Snapshot: MvccSnapshot[applied={T|T < 7334725234022993920}]
I20260929 17:53:22.849596 4994 mvcc.cc:205] Tried to move back new op lower bound from 7334725234022993920 to 7334725233871536128. Current Snapshot: MvccSnapshot[applied={T|T < 7334725234022993920}]
I20260929 17:53:22.856774 4506 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53604:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 17:53:22.856940 4506 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53604:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 17:53:22.859196 4506 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f6966161f41a458e91ad757e8ac08b49: Sending DeleteTablet for 3 replicas of tablet 60a86e20980841d29150e56819d942bd
I20260929 17:53:22.859923 4630 tablet_service.cc:1514] Processing DeleteTablet for tablet 60a86e20980841d29150e56819d942bd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 17:53:22 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36434
I20260929 17:53:22.860046 4896 tablet_service.cc:1514] Processing DeleteTablet for tablet 60a86e20980841d29150e56819d942bd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 17:53:22 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38186
I20260929 17:53:22.860188 4763 tablet_service.cc:1514] Processing DeleteTablet for tablet 60a86e20980841d29150e56819d942bd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 17:53:22 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51212
I20260929 17:53:22.860556 5003 tablet_replica.cc:330] T 60a86e20980841d29150e56819d942bd P ee58869ee79d4beba9a998d90ac93364: stopping tablet replica
I20260929 17:53:22.860685 5002 tablet_replica.cc:330] T 60a86e20980841d29150e56819d942bd P aa267a00eaed4a448e0dadf7edbc044a: stopping tablet replica
I20260929 17:53:22.860868 5003 raft_consensus.cc:2229] T 60a86e20980841d29150e56819d942bd P ee58869ee79d4beba9a998d90ac93364 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 17:53:22.860900 5002 raft_consensus.cc:2229] T 60a86e20980841d29150e56819d942bd P aa267a00eaed4a448e0dadf7edbc044a [term 1 LEADER]: Raft consensus shutting down.
I20260929 17:53:22.861162 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskup3Soe/build/debug/bin/kudu with pid 4568
I20260929 17:53:22.861313 5003 raft_consensus.cc:2258] T 60a86e20980841d29150e56819d942bd P ee58869ee79d4beba9a998d90ac93364 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 17:53:22.861384 5004 tablet_replica.cc:330] T 60a86e20980841d29150e56819d942bd P 183882187043422882fa96e52130ed82: stopping tablet replica
I20260929 17:53:22.861825 5004 raft_consensus.cc:2229] T 60a86e20980841d29150e56819d942bd P 183882187043422882fa96e52130ed82 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 17:53:22.862079 5004 raft_consensus.cc:2258] T 60a86e20980841d29150e56819d942bd P 183882187043422882fa96e52130ed82 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 17:53:22.863775 5003 ts_tablet_manager.cc:1905] T 60a86e20980841d29150e56819d942bd P ee58869ee79d4beba9a998d90ac93364: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 17:53:22.864306 5004 ts_tablet_manager.cc:1905] T 60a86e20980841d29150e56819d942bd P 183882187043422882fa96e52130ed82: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 17:53:22.866762 5003 ts_tablet_manager.cc:1918] T 60a86e20980841d29150e56819d942bd P ee58869ee79d4beba9a998d90ac93364: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 17:53:22.866868 5003 log.cc:1196] T 60a86e20980841d29150e56819d942bd P ee58869ee79d4beba9a998d90ac93364: Deleting WAL directory at /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704375975685-392-0/minicluster-data/ts-2/wal/wals/60a86e20980841d29150e56819d942bd
I20260929 17:53:22.866968 5004 ts_tablet_manager.cc:1918] T 60a86e20980841d29150e56819d942bd P 183882187043422882fa96e52130ed82: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 17:53:22.867055 5004 log.cc:1196] T 60a86e20980841d29150e56819d942bd P 183882187043422882fa96e52130ed82: Deleting WAL directory at /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790704375975685-392-0/minicluster-data/ts-1/wal/wals/60a86e20980841d29150e56819d942bd
I20260929 17:53:22.867331 5004 ts_tablet_manager.cc:1939] T 60a86e20980841d29150e56819d942bd P 183882187043422882fa96e52130ed82: Deleting consensus metadata
I20260929 17:53:22.867497 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskup3Soe/build/debug/bin/kudu with pid 4701
W20260929 17:53:22.867643 4492 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:33405 (error 108)
I20260929 17:53:22.868770 4975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:22.868489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33405 (local address 127.0.0.1:36454)
0929 17:53:22.868611 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:22.868678 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260929 17:53:22.868937 4492 catalog_manager.cc:4584] TS aa267a00eaed4a448e0dadf7edbc044a (127.0.98.1:33405): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 60a86e20980841d29150e56819d942bd: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33405: connect: Connection refused (error 111)
I20260929 17:53:22.869287 5003 ts_tablet_manager.cc:1939] T 60a86e20980841d29150e56819d942bd P ee58869ee79d4beba9a998d90ac93364: Deleting consensus metadata
I20260929 17:53:22.870391 4493 catalog_manager.cc:4841] TS ee58869ee79d4beba9a998d90ac93364 (127.0.98.3:34105): tablet 60a86e20980841d29150e56819d942bd (table test-table [id=0f30ae76db8c494ea59f60bf1ac95c8e]) successfully deleted
I20260929 17:53:22.872668 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskup3Soe/build/debug/bin/kudu with pid 4834
I20260929 17:53:22.877280 4975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:22.872724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41173 (local address 127.0.0.1:51222)
0929 17:53:22.877142 (+ 4418us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:22.877195 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:41173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":4324}
I20260929 17:53:22.877386 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskup3Soe/build/debug/bin/kudu with pid 4475
2026-09-29T17:53:22Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2195 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-29T17:53:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T17:53:22Z Disabled control of system clock
I20260929 17:53:22.904202 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskup3Soe/build/debug/bin/kudu
/tmp/dist-test-taskup3Soe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790704375975685-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790704375975685-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790704375975685-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790704375975685-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40323
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:34161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40323
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260929 17:53:23.009694 5009 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:53:23.009954 5009 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:53:23.009989 5009 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:53:23.013885 5009 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 17:53:23.013944 5009 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:53:23.013958 5009 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:53:23.013973 5009 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 17:53:23.013988 5009 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 17:53:23.018782 5009 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790704375975685-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790704375975685-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40323
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40323
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790704375975685-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5009
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790704375975685-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:53:23.020023 5009 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:53:23.021844 5009 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:53:23.030687 5015 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:53:23.030763 5009 server_base.cc:1034] running on GCE node
W20260929 17:53:23.030702 5014 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:53:23.030922 5017 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:53:23.031519 5009 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:53:23.032588 5009 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:53:23.033738 5009 hybrid_clock.cc:648] HybridClock initialized: now 1790704403033723 us; error 26 us; skew 500 ppm
I20260929 17:53:23.035475 5009 webserver.cc:460] Webserver started at http://127.0.98.62:44971/ using document root <none> and password file <none>
I20260929 17:53:23.036005 5009 fs_manager.cc:359] Metadata directory not provided
I20260929 17:53:23.036069 5009 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:53:23.036255 5009 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:53:23.038260 5009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790704375975685-392-0/minicluster-data/master-0/data/instance:
uuid: "8b46676206024b629b69b10115928a41"
format_stamp: "Formatted at 2026-09-29 17:53:23 on dist-test-slave-53nh"
server_key: "ccb0780c02f9f0294eacfbd690d2ecd2"
server_key_iv: "3aa2cd163a3b94023e6c37868fb95b39"
server_key_version: "encryptionkey@0"
I20260929 17:53:23.038793 5009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790704375975685-392-0/minicluster-data/master-0/wal/instance:
uuid: "8b46676206024b629b69b10115928a41"
format_stamp: "Formatted at 2026-09-29 17:53:23 on dist-test-slave-53nh"
server_key: "ccb0780c02f9f0294eacfbd690d2ecd2"
server_key_iv: "3aa2cd163a3b94023e6c37868fb95b39"
server_key_version: "encryptionkey@0"
I20260929 17:53:23.042232 5009 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 17:53:23.044399 5023 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:53:23.045521 5009 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 17:53:23.045626 5009 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790704375975685-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790704375975685-392-0/minicluster-data/master-0/wal
uuid: "8b46676206024b629b69b10115928a41"
format_stamp: "Formatted at 2026-09-29 17:53:23 on dist-test-slave-53nh"
server_key: "ccb0780c02f9f0294eacfbd690d2ecd2"
server_key_iv: "3aa2cd163a3b94023e6c37868fb95b39"
server_key_version: "encryptionkey@0"
I20260929 17:53:23.045717 5009 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790704375975685-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790704375975685-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790704375975685-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:53:23.049959 5009 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:53:23.050617 5009 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:53:23.050776 5009 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:53:23.056833 5009 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40323
I20260929 17:53:23.056834 5075 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40323 every 8 connection(s)
I20260929 17:53:23.057739 5009 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790704375975685-392-0/minicluster-data/master-0/data/info.pb
I20260929 17:53:23.059095 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskup3Soe/build/debug/bin/kudu as pid 5009
I20260929 17:53:23.059219 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790704375975685-392-0/minicluster-data/master-0/wal/instance
I20260929 17:53:23.059510 392 external_mini_cluster.cc:1428] Setting key e69a522628d3da036486d1fcbaf8c6f8
I20260929 17:53:23.061136 5076 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:53:23.064877 5079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:23.060852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56914 (local address 127.0.98.62:40323)
0929 17:53:23.061596 (+ 744us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:23.061611 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:23.061644 (+ 33us) server_negotiation.cc:407] Connection header received
0929 17:53:23.062262 (+ 618us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:23.062276 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:23.062616 (+ 340us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:23.062912 (+ 296us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:53:23.063688 (+ 776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:53:23.063692 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:53:23.063705 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 17:53:23.063715 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:53:23.063768 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 17:53:23.063784 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:53:23.063920 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0929 17:53:23.063989 (+ 69us) server_negotiation.cc:299] Negotiation successful
0929 17:53:23.064221 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":356,"thread_start_us":145,"threads_started":1}
I20260929 17:53:23.066488 5076 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b46676206024b629b69b10115928a41: Bootstrap starting.
I20260929 17:53:23.068605 5076 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8b46676206024b629b69b10115928a41: Neither blocks nor log segments found. Creating new log.
I20260929 17:53:23.069299 5076 log.cc:824] T 00000000000000000000000000000000 P 8b46676206024b629b69b10115928a41: Log is configured to *not* fsync() on all Append() calls
I20260929 17:53:23.070925 5076 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b46676206024b629b69b10115928a41: No bootstrap required, opened a new log
I20260929 17:53:23.074203 5076 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8b46676206024b629b69b10115928a41 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b46676206024b629b69b10115928a41" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40323 } }
I20260929 17:53:23.074409 5076 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8b46676206024b629b69b10115928a41 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:53:23.074471 5076 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8b46676206024b629b69b10115928a41 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b46676206024b629b69b10115928a41, State: Initialized, Role: FOLLOWER
I20260929 17:53:23.074890 5076 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8b46676206024b629b69b10115928a41 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b46676206024b629b69b10115928a41" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40323 } }
I20260929 17:53:23.075052 5076 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8b46676206024b629b69b10115928a41 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 17:53:23.075120 5076 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8b46676206024b629b69b10115928a41 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 17:53:23.075217 5076 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8b46676206024b629b69b10115928a41 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:53:23.076086 5076 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8b46676206024b629b69b10115928a41 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b46676206024b629b69b10115928a41" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40323 } }
I20260929 17:53:23.076396 5076 leader_election.cc:302] T 00000000000000000000000000000000 P 8b46676206024b629b69b10115928a41 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8b46676206024b629b69b10115928a41; no voters:
I20260929 17:53:23.076658 5076 leader_election.cc:288] T 00000000000000000000000000000000 P 8b46676206024b629b69b10115928a41 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 17:53:23.076756 5081 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8b46676206024b629b69b10115928a41 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 17:53:23.076972 5081 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8b46676206024b629b69b10115928a41 [term 1 LEADER]: Becoming Leader. State: Replica: 8b46676206024b629b69b10115928a41, State: Running, Role: LEADER
I20260929 17:53:23.077284 5081 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8b46676206024b629b69b10115928a41 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b46676206024b629b69b10115928a41" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40323 } }
I20260929 17:53:23.077775 5076 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8b46676206024b629b69b10115928a41 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 17:53:23.078794 5083 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b46676206024b629b69b10115928a41 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8b46676206024b629b69b10115928a41. Latest consensus state: current_term: 1 leader_uuid: "8b46676206024b629b69b10115928a41" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b46676206024b629b69b10115928a41" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40323 } } }
I20260929 17:53:23.078917 5083 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b46676206024b629b69b10115928a41 [sys.catalog]: This master's current role is: LEADER
I20260929 17:53:23.079159 5082 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b46676206024b629b69b10115928a41 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8b46676206024b629b69b10115928a41" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b46676206024b629b69b10115928a41" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40323 } } }
I20260929 17:53:23.079262 5082 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b46676206024b629b69b10115928a41 [sys.catalog]: This master's current role is: LEADER
I20260929 17:53:23.079334 5090 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 17:53:23.082193 5090 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 17:53:23.087884 5090 catalog_manager.cc:1347] Generated new cluster ID: 4891d13f1d2c4f6481271b6eadc9d111
I20260929 17:53:23.087944 5090 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 17:53:23.108028 5090 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 17:53:23.108346 5090 catalog_manager.cc:1504] Loading token signing keys...
I20260929 17:53:23.113273 5090 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8b46676206024b629b69b10115928a41: Generated new TSK 0
I20260929 17:53:23.114006 5090 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 17:53:23.127287 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskup3Soe/build/debug/bin/kudu
/tmp/dist-test-taskup3Soe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790704375975685-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790704375975685-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790704375975685-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790704375975685-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40323
--builtin_ntp_servers=127.0.98.20:34161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260929 17:53:23.234709 5100 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:53:23.234979 5100 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:53:23.235036 5100 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:53:23.239056 5100 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:53:23.239122 5100 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:53:23.239200 5100 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 17:53:23.243992 5100 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790704375975685-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790704375975685-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790704375975685-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40323
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5100
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790704375975685-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:53:23.245111 5100 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:53:23.246471 5100 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:53:23.256019 5105 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:53:23.256108 5100 server_base.cc:1034] running on GCE node
W20260929 17:53:23.256027 5106 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:53:23.256014 5108 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:53:23.256649 5100 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:53:23.257357 5100 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:53:23.258518 5100 hybrid_clock.cc:648] HybridClock initialized: now 1790704403258500 us; error 31 us; skew 500 ppm
I20260929 17:53:23.260288 5100 webserver.cc:460] Webserver started at http://127.0.98.1:44449/ using document root <none> and password file <none>
I20260929 17:53:23.260845 5100 fs_manager.cc:359] Metadata directory not provided
I20260929 17:53:23.260911 5100 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:53:23.261140 5100 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:53:23.263129 5100 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790704375975685-392-0/minicluster-data/ts-0/data/instance:
uuid: "ea3db7c9c7e849318988ddc7be9ab05c"
format_stamp: "Formatted at 2026-09-29 17:53:23 on dist-test-slave-53nh"
server_key: "fba8dc225ff0181bdeb5dc0164e40ad6"
server_key_iv: "3e0050fbede5f325415c613be6c19114"
server_key_version: "encryptionkey@0"
I20260929 17:53:23.263732 5100 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790704375975685-392-0/minicluster-data/ts-0/wal/instance:
uuid: "ea3db7c9c7e849318988ddc7be9ab05c"
format_stamp: "Formatted at 2026-09-29 17:53:23 on dist-test-slave-53nh"
server_key: "fba8dc225ff0181bdeb5dc0164e40ad6"
server_key_iv: "3e0050fbede5f325415c613be6c19114"
server_key_version: "encryptionkey@0"
I20260929 17:53:23.267287 5100 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 17:53:23.269546 5114 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:53:23.270691 5100 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 17:53:23.270812 5100 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790704375975685-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790704375975685-392-0/minicluster-data/ts-0/wal
uuid: "ea3db7c9c7e849318988ddc7be9ab05c"
format_stamp: "Formatted at 2026-09-29 17:53:23 on dist-test-slave-53nh"
server_key: "fba8dc225ff0181bdeb5dc0164e40ad6"
server_key_iv: "3e0050fbede5f325415c613be6c19114"
server_key_version: "encryptionkey@0"
I20260929 17:53:23.270913 5100 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790704375975685-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790704375975685-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790704375975685-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:53:23.275398 5100 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:53:23.275972 5100 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:53:23.276117 5100 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:53:23.276623 5100 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 17:53:23.277545 5100 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:53:23.277590 5100 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:53:23.277629 5100 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:53:23.277657 5100 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 17:53:23.285784 5100 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34223
I20260929 17:53:23.285804 5227 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34223 every 8 connection(s)
I20260929 17:53:23.286643 5100 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790704375975685-392-0/minicluster-data/ts-0/data/info.pb
I20260929 17:53:23.292320 5079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:23.288302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59201 (local address 127.0.98.62:40323)
0929 17:53:23.288428 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:23.288432 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:23.289534 (+ 1102us) server_negotiation.cc:407] Connection header received
0929 17:53:23.290776 (+ 1242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:23.290779 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:23.290822 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:23.290898 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:53:23.291909 (+ 1011us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:53:23.291912 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:53:23.291915 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 17:53:23.291927 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:53:23.291949 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 17:53:23.291956 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:53:23.292030 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0929 17:53:23.292145 (+ 115us) server_negotiation.cc:299] Negotiation successful
0929 17:53:23.292187 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 17:53:23.292896 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskup3Soe/build/debug/bin/kudu as pid 5100
I20260929 17:53:23.293030 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790704375975685-392-0/minicluster-data/ts-0/wal/instance
I20260929 17:53:23.293329 392 external_mini_cluster.cc:1428] Setting key d182f60875da3231f49ff62b4ece20fc
I20260929 17:53:23.293339 5230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:23.288698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40323 (local address 127.0.98.1:59201)
0929 17:53:23.289403 (+ 705us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:23.289440 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0929 17:53:23.290575 (+ 1135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:53:23.291058 (+ 483us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:53:23.291065 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:53:23.291147 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:53:23.291667 (+ 520us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 17:53:23.291669 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:53:23.291708 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 17:53:23.291775 (+ 67us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:53:23.292023 (+ 248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:53:23.292030 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:53:23.292035 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 17:53:23.292180 (+ 145us) client_negotiation.cc:240] Negotiation successful
0929 17:53:23.292417 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":123,"threads_started":1}
I20260929 17:53:23.294739 5228 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40323
I20260929 17:53:23.294976 5228 heartbeater.cc:464] Registering TS with master...
I20260929 17:53:23.295421 5228 heartbeater.cc:511] Master 127.0.98.62:40323 requested a full tablet report, sending...
I20260929 17:53:23.295600 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskup3Soe/build/debug/bin/kudu
/tmp/dist-test-taskup3Soe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790704375975685-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790704375975685-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790704375975685-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790704375975685-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40323
--builtin_ntp_servers=127.0.98.20:34161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260929 17:53:23.297129 5039 ts_manager.cc:194] Registered new tserver with Master: ea3db7c9c7e849318988ddc7be9ab05c (127.0.98.1:34223)
W20260929 17:53:23.402774 5231 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:53:23.403064 5231 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:53:23.403129 5231 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:53:23.407250 5231 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:53:23.407320 5231 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:53:23.407403 5231 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 17:53:23.412218 5231 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790704375975685-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790704375975685-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790704375975685-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40323
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5231
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790704375975685-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:53:23.413326 5231 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:53:23.414698 5231 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:53:23.424032 5237 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:53:23.424046 5236 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:53:23.424055 5239 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:53:23.424453 5231 server_base.cc:1034] running on GCE node
I20260929 17:53:23.424853 5231 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:53:23.425529 5231 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:53:23.426724 5231 hybrid_clock.cc:648] HybridClock initialized: now 1790704403426699 us; error 41 us; skew 500 ppm
I20260929 17:53:23.428769 5231 webserver.cc:460] Webserver started at http://127.0.98.2:46087/ using document root <none> and password file <none>
I20260929 17:53:23.429404 5231 fs_manager.cc:359] Metadata directory not provided
I20260929 17:53:23.429474 5231 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:53:23.429678 5231 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:53:23.431811 5231 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790704375975685-392-0/minicluster-data/ts-1/data/instance:
uuid: "f040f18044544356a61ff6d7b60f15da"
format_stamp: "Formatted at 2026-09-29 17:53:23 on dist-test-slave-53nh"
server_key: "ab5cc7656dca84f771144f30b9f3bad2"
server_key_iv: "6c64f11e8f619b67431e523f9fb7035c"
server_key_version: "encryptionkey@0"
I20260929 17:53:23.432446 5231 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790704375975685-392-0/minicluster-data/ts-1/wal/instance:
uuid: "f040f18044544356a61ff6d7b60f15da"
format_stamp: "Formatted at 2026-09-29 17:53:23 on dist-test-slave-53nh"
server_key: "ab5cc7656dca84f771144f30b9f3bad2"
server_key_iv: "6c64f11e8f619b67431e523f9fb7035c"
server_key_version: "encryptionkey@0"
I20260929 17:53:23.436355 5231 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 17:53:23.438856 5245 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:53:23.440105 5231 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260929 17:53:23.440258 5231 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790704375975685-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790704375975685-392-0/minicluster-data/ts-1/wal
uuid: "f040f18044544356a61ff6d7b60f15da"
format_stamp: "Formatted at 2026-09-29 17:53:23 on dist-test-slave-53nh"
server_key: "ab5cc7656dca84f771144f30b9f3bad2"
server_key_iv: "6c64f11e8f619b67431e523f9fb7035c"
server_key_version: "encryptionkey@0"
I20260929 17:53:23.440361 5231 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790704375975685-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790704375975685-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790704375975685-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:53:23.444818 5231 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:53:23.445448 5231 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:53:23.445600 5231 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:53:23.446115 5231 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 17:53:23.446974 5231 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:53:23.447018 5231 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:53:23.447067 5231 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:53:23.447093 5231 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 17:53:23.455076 5231 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34325
I20260929 17:53:23.455130 5358 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34325 every 8 connection(s)
I20260929 17:53:23.455929 5231 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790704375975685-392-0/minicluster-data/ts-1/data/info.pb
I20260929 17:53:23.461305 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskup3Soe/build/debug/bin/kudu as pid 5231
I20260929 17:53:23.461438 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790704375975685-392-0/minicluster-data/ts-1/wal/instance
I20260929 17:53:23.461742 392 external_mini_cluster.cc:1428] Setting key 8176ed4f47e0aedd5b3e651a93d990f8
I20260929 17:53:23.461964 5079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:23.457698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55711 (local address 127.0.98.62:40323)
0929 17:53:23.457862 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:23.457866 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:23.458942 (+ 1076us) server_negotiation.cc:407] Connection header received
0929 17:53:23.460182 (+ 1240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:23.460185 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:23.460237 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:23.460310 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:53:23.461498 (+ 1188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:53:23.461502 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:53:23.461505 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 17:53:23.461519 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:53:23.461543 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 17:53:23.461552 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:53:23.461631 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0929 17:53:23.461815 (+ 184us) server_negotiation.cc:299] Negotiation successful
0929 17:53:23.461859 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260929 17:53:23.462805 5361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:23.458103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40323 (local address 127.0.98.2:55711)
0929 17:53:23.458795 (+ 692us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:23.458834 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 17:53:23.459985 (+ 1151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:53:23.460494 (+ 509us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:53:23.460501 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:53:23.460594 (+ 93us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:53:23.461184 (+ 590us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 17:53:23.461186 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:53:23.461240 (+ 54us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 17:53:23.461329 (+ 89us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:53:23.461635 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:53:23.461644 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:53:23.461650 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 17:53:23.461852 (+ 202us) client_negotiation.cc:240] Negotiation successful
0929 17:53:23.462090 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":120,"threads_started":1}
I20260929 17:53:23.463647 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskup3Soe/build/debug/bin/kudu
/tmp/dist-test-taskup3Soe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790704375975685-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790704375975685-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790704375975685-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790704375975685-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40323
--builtin_ntp_servers=127.0.98.20:34161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260929 17:53:23.463899 5359 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40323
I20260929 17:53:23.464125 5359 heartbeater.cc:464] Registering TS with master...
I20260929 17:53:23.464516 5359 heartbeater.cc:511] Master 127.0.98.62:40323 requested a full tablet report, sending...
I20260929 17:53:23.465674 5039 ts_manager.cc:194] Registered new tserver with Master: f040f18044544356a61ff6d7b60f15da (127.0.98.2:34325)
W20260929 17:53:23.572108 5362 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:53:23.572386 5362 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:53:23.572427 5362 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:53:23.576576 5362 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:53:23.576650 5362 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:53:23.576713 5362 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 17:53:23.581681 5362 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790704375975685-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790704375975685-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790704375975685-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40323
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5362
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790704375975685-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:53:23.582720 5362 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:53:23.584062 5362 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:53:23.593174 5368 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:53:23.593182 5370 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:53:23.593182 5367 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:53:23.593439 5362 server_base.cc:1034] running on GCE node
I20260929 17:53:23.593861 5362 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:53:23.594481 5362 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:53:23.595636 5362 hybrid_clock.cc:648] HybridClock initialized: now 1790704403595611 us; error 37 us; skew 500 ppm
I20260929 17:53:23.597455 5362 webserver.cc:460] Webserver started at http://127.0.98.3:46265/ using document root <none> and password file <none>
I20260929 17:53:23.598008 5362 fs_manager.cc:359] Metadata directory not provided
I20260929 17:53:23.598074 5362 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:53:23.598253 5362 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:53:23.600256 5362 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790704375975685-392-0/minicluster-data/ts-2/data/instance:
uuid: "7438304ccbf84631897c0b06fdfcfcc2"
format_stamp: "Formatted at 2026-09-29 17:53:23 on dist-test-slave-53nh"
server_key: "35bf99233b20b9d166ade10b347d673c"
server_key_iv: "5aedb48fe7d3761930c472e6e1970e13"
server_key_version: "encryptionkey@0"
I20260929 17:53:23.600766 5362 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790704375975685-392-0/minicluster-data/ts-2/wal/instance:
uuid: "7438304ccbf84631897c0b06fdfcfcc2"
format_stamp: "Formatted at 2026-09-29 17:53:23 on dist-test-slave-53nh"
server_key: "35bf99233b20b9d166ade10b347d673c"
server_key_iv: "5aedb48fe7d3761930c472e6e1970e13"
server_key_version: "encryptionkey@0"
I20260929 17:53:23.604260 5362 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260929 17:53:23.606583 5376 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:53:23.607728 5362 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 17:53:23.607838 5362 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790704375975685-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790704375975685-392-0/minicluster-data/ts-2/wal
uuid: "7438304ccbf84631897c0b06fdfcfcc2"
format_stamp: "Formatted at 2026-09-29 17:53:23 on dist-test-slave-53nh"
server_key: "35bf99233b20b9d166ade10b347d673c"
server_key_iv: "5aedb48fe7d3761930c472e6e1970e13"
server_key_version: "encryptionkey@0"
I20260929 17:53:23.607930 5362 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790704375975685-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790704375975685-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790704375975685-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:53:23.612380 5362 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:53:23.612950 5362 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:53:23.613134 5362 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:53:23.613662 5362 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 17:53:23.614521 5362 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:53:23.614566 5362 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:53:23.614614 5362 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:53:23.614639 5362 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 17:53:23.622630 5362 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42171
I20260929 17:53:23.622673 5489 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42171 every 8 connection(s)
I20260929 17:53:23.623482 5362 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790704375975685-392-0/minicluster-data/ts-2/data/info.pb
I20260929 17:53:23.629216 5079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:23.625170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36629 (local address 127.0.98.62:40323)
0929 17:53:23.625323 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:23.625327 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:23.626385 (+ 1058us) server_negotiation.cc:407] Connection header received
0929 17:53:23.627667 (+ 1282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:23.627671 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:23.627714 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:23.627795 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:53:23.628776 (+ 981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:53:23.628779 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:53:23.628782 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 17:53:23.628795 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:53:23.628816 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 17:53:23.628825 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:53:23.628897 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0929 17:53:23.629034 (+ 137us) server_negotiation.cc:299] Negotiation successful
0929 17:53:23.629077 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260929 17:53:23.629619 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskup3Soe/build/debug/bin/kudu as pid 5362
I20260929 17:53:23.629729 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790704375975685-392-0/minicluster-data/ts-2/wal/instance
I20260929 17:53:23.629964 392 external_mini_cluster.cc:1428] Setting key 1f95b309110a93fb4c87cb211e574d16
I20260929 17:53:23.630064 5492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:23.625595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40323 (local address 127.0.98.3:36629)
0929 17:53:23.626241 (+ 646us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:23.626282 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 17:53:23.627465 (+ 1183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:53:23.627959 (+ 494us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:53:23.627966 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:53:23.628054 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:53:23.628541 (+ 487us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 17:53:23.628543 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:53:23.628580 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 17:53:23.628644 (+ 64us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:53:23.628890 (+ 246us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:53:23.628897 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:53:23.628902 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 17:53:23.629075 (+ 173us) client_negotiation.cc:240] Negotiation successful
0929 17:53:23.629304 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":106,"threads_started":1}
I20260929 17:53:23.631098 5490 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40323
I20260929 17:53:23.631304 5490 heartbeater.cc:464] Registering TS with master...
I20260929 17:53:23.631666 5490 heartbeater.cc:511] Master 127.0.98.62:40323 requested a full tablet report, sending...
I20260929 17:53:23.632534 5039 ts_manager.cc:194] Registered new tserver with Master: 7438304ccbf84631897c0b06fdfcfcc2 (127.0.98.3:42171)
I20260929 17:53:23.643477 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 17:53:23.646694 5079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:23.645354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56930 (local address 127.0.98.62:40323)
0929 17:53:23.645471 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:23.645475 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:23.645616 (+ 141us) server_negotiation.cc:407] Connection header received
0929 17:53:23.645725 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:23.645728 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:23.645761 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:23.645822 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:53:23.646409 (+ 587us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:53:23.646412 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:53:23.646414 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 17:53:23.646424 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:53:23.646447 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 17:53:23.646458 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:53:23.646518 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0929 17:53:23.646565 (+ 47us) server_negotiation.cc:299] Negotiation successful
0929 17:53:23.646595 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260929 17:53:23.649287 5039 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56930:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 17:53:23.652216 5039 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 17:53:23.663583 5504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:23.660709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34325 (local address 127.0.0.1:48140)
0929 17:53:23.661740 (+ 1031us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:23.661752 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 17:53:23.661824 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:53:23.662364 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:53:23.662367 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:53:23.662382 (+ 15us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:53:23.662982 (+ 600us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 17:53:23.662983 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:53:23.662991 (+ 8us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 17:53:23.663007 (+ 16us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:53:23.663268 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:53:23.663270 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:53:23.663273 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 17:53:23.663356 (+ 83us) client_negotiation.cc:240] Negotiation successful
0929 17:53:23.663422 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":972,"thread_start_us":128,"threads_started":1}
I20260929 17:53:23.663640 5503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:23.660730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40430 (local address 127.0.98.1:34223)
0929 17:53:23.661348 (+ 618us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:23.661360 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:23.661660 (+ 300us) server_negotiation.cc:407] Connection header received
0929 17:53:23.661723 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:23.661729 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:23.661930 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:23.662454 (+ 524us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:53:23.663073 (+ 619us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:53:23.663079 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:53:23.663091 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 17:53:23.663101 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:53:23.663140 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 17:53:23.663166 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:53:23.663272 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0929 17:53:23.663399 (+ 127us) server_negotiation.cc:299] Negotiation successful
0929 17:53:23.663472 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":514,"thread_start_us":155,"threads_started":1}
I20260929 17:53:23.663583 5502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:23.660559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34223 (local address 127.0.0.1:40430)
0929 17:53:23.661437 (+ 878us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:23.661485 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0929 17:53:23.661579 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:53:23.662100 (+ 521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:53:23.662106 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:53:23.662181 (+ 75us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:53:23.662758 (+ 577us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 17:53:23.662760 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:53:23.662780 (+ 20us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 17:53:23.662856 (+ 76us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:53:23.663253 (+ 397us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:53:23.663262 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:53:23.663268 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 17:53:23.663371 (+ 103us) client_negotiation.cc:240] Negotiation successful
0929 17:53:23.663422 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":678,"thread_start_us":82,"threads_started":1}
I20260929 17:53:23.664373 5506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:23.661371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48140 (local address 127.0.98.2:34325)
0929 17:53:23.662018 (+ 647us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:23.662029 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:23.662051 (+ 22us) server_negotiation.cc:407] Connection header received
0929 17:53:23.662099 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:23.662102 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:23.662233 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:23.662358 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:53:23.663103 (+ 745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:53:23.663108 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:53:23.663116 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 17:53:23.663125 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:53:23.663160 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 17:53:23.663182 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:53:23.664081 (+ 899us) server_negotiation.cc:1035] Waiting for connection context
0929 17:53:23.664129 (+ 48us) server_negotiation.cc:299] Negotiation successful
0929 17:53:23.664200 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":552,"thread_start_us":102,"threads_started":1}
I20260929 17:53:23.664546 5501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:23.660965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42171 (local address 127.0.0.1:46298)
0929 17:53:23.661963 (+ 998us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:23.661976 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 17:53:23.662034 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:53:23.663473 (+ 1439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:53:23.663476 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:53:23.663489 (+ 13us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:53:23.663944 (+ 455us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 17:53:23.663946 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:53:23.663957 (+ 11us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 17:53:23.663986 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:53:23.664332 (+ 346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:53:23.664335 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:53:23.664338 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 17:53:23.664398 (+ 60us) client_negotiation.cc:240] Negotiation successful
0929 17:53:23.664430 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":950,"thread_start_us":109,"threads_started":1}
I20260929 17:53:23.665092 5505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:23.661374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46298 (local address 127.0.98.3:42171)
0929 17:53:23.661832 (+ 458us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:23.661843 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:23.662113 (+ 270us) server_negotiation.cc:407] Connection header received
0929 17:53:23.662164 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:23.662169 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:23.662342 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:23.662955 (+ 613us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:53:23.664112 (+ 1157us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:53:23.664118 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:53:23.664129 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 17:53:23.664140 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:53:23.664176 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 17:53:23.664201 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:53:23.664291 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0929 17:53:23.664836 (+ 545us) server_negotiation.cc:299] Negotiation successful
0929 17:53:23.664903 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":360,"thread_start_us":99,"threads_started":1}
I20260929 17:53:23.667045 5162 tablet_service.cc:1467] Processing CreateTablet for tablet e1ec5801ccef46058c0814de78ba3b79 (DEFAULT_TABLE table=test-table [id=9200e80e936e4de98b5f1dee3fe4ff29]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 17:53:23.667321 5293 tablet_service.cc:1467] Processing CreateTablet for tablet e1ec5801ccef46058c0814de78ba3b79 (DEFAULT_TABLE table=test-table [id=9200e80e936e4de98b5f1dee3fe4ff29]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 17:53:23.667665 5424 tablet_service.cc:1467] Processing CreateTablet for tablet e1ec5801ccef46058c0814de78ba3b79 (DEFAULT_TABLE table=test-table [id=9200e80e936e4de98b5f1dee3fe4ff29]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 17:53:23.668171 5162 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1ec5801ccef46058c0814de78ba3b79. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:53:23.668174 5293 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1ec5801ccef46058c0814de78ba3b79. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:53:23.668541 5424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1ec5801ccef46058c0814de78ba3b79. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:53:23.674127 5507 tablet_bootstrap.cc:492] T e1ec5801ccef46058c0814de78ba3b79 P f040f18044544356a61ff6d7b60f15da: Bootstrap starting.
I20260929 17:53:23.675732 5508 tablet_bootstrap.cc:492] T e1ec5801ccef46058c0814de78ba3b79 P 7438304ccbf84631897c0b06fdfcfcc2: Bootstrap starting.
I20260929 17:53:23.675755 5509 tablet_bootstrap.cc:492] T e1ec5801ccef46058c0814de78ba3b79 P ea3db7c9c7e849318988ddc7be9ab05c: Bootstrap starting.
I20260929 17:53:23.676461 5507 tablet_bootstrap.cc:654] T e1ec5801ccef46058c0814de78ba3b79 P f040f18044544356a61ff6d7b60f15da: Neither blocks nor log segments found. Creating new log.
I20260929 17:53:23.677384 5507 log.cc:824] T e1ec5801ccef46058c0814de78ba3b79 P f040f18044544356a61ff6d7b60f15da: Log is configured to *not* fsync() on all Append() calls
I20260929 17:53:23.677541 5509 tablet_bootstrap.cc:654] T e1ec5801ccef46058c0814de78ba3b79 P ea3db7c9c7e849318988ddc7be9ab05c: Neither blocks nor log segments found. Creating new log.
I20260929 17:53:23.677824 5508 tablet_bootstrap.cc:654] T e1ec5801ccef46058c0814de78ba3b79 P 7438304ccbf84631897c0b06fdfcfcc2: Neither blocks nor log segments found. Creating new log.
I20260929 17:53:23.678429 5509 log.cc:824] T e1ec5801ccef46058c0814de78ba3b79 P ea3db7c9c7e849318988ddc7be9ab05c: Log is configured to *not* fsync() on all Append() calls
I20260929 17:53:23.678655 5508 log.cc:824] T e1ec5801ccef46058c0814de78ba3b79 P 7438304ccbf84631897c0b06fdfcfcc2: Log is configured to *not* fsync() on all Append() calls
I20260929 17:53:23.679682 5507 tablet_bootstrap.cc:492] T e1ec5801ccef46058c0814de78ba3b79 P f040f18044544356a61ff6d7b60f15da: No bootstrap required, opened a new log
I20260929 17:53:23.679874 5507 ts_tablet_manager.cc:1397] T e1ec5801ccef46058c0814de78ba3b79 P f040f18044544356a61ff6d7b60f15da: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260929 17:53:23.680387 5509 tablet_bootstrap.cc:492] T e1ec5801ccef46058c0814de78ba3b79 P ea3db7c9c7e849318988ddc7be9ab05c: No bootstrap required, opened a new log
I20260929 17:53:23.680584 5509 ts_tablet_manager.cc:1397] T e1ec5801ccef46058c0814de78ba3b79 P ea3db7c9c7e849318988ddc7be9ab05c: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260929 17:53:23.681094 5508 tablet_bootstrap.cc:492] T e1ec5801ccef46058c0814de78ba3b79 P 7438304ccbf84631897c0b06fdfcfcc2: No bootstrap required, opened a new log
I20260929 17:53:23.681262 5508 ts_tablet_manager.cc:1397] T e1ec5801ccef46058c0814de78ba3b79 P 7438304ccbf84631897c0b06fdfcfcc2: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 17:53:23.684412 5507 raft_consensus.cc:348] T e1ec5801ccef46058c0814de78ba3b79 P f040f18044544356a61ff6d7b60f15da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea3db7c9c7e849318988ddc7be9ab05c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34223 } } peers { permanent_uuid: "f040f18044544356a61ff6d7b60f15da" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34325 } } peers { permanent_uuid: "7438304ccbf84631897c0b06fdfcfcc2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42171 } }
I20260929 17:53:23.684487 5508 raft_consensus.cc:348] T e1ec5801ccef46058c0814de78ba3b79 P 7438304ccbf84631897c0b06fdfcfcc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea3db7c9c7e849318988ddc7be9ab05c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34223 } } peers { permanent_uuid: "f040f18044544356a61ff6d7b60f15da" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34325 } } peers { permanent_uuid: "7438304ccbf84631897c0b06fdfcfcc2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42171 } }
I20260929 17:53:23.684681 5507 raft_consensus.cc:374] T e1ec5801ccef46058c0814de78ba3b79 P f040f18044544356a61ff6d7b60f15da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:53:23.684679 5508 raft_consensus.cc:374] T e1ec5801ccef46058c0814de78ba3b79 P 7438304ccbf84631897c0b06fdfcfcc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:53:23.684738 5508 raft_consensus.cc:729] T e1ec5801ccef46058c0814de78ba3b79 P 7438304ccbf84631897c0b06fdfcfcc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7438304ccbf84631897c0b06fdfcfcc2, State: Initialized, Role: FOLLOWER
I20260929 17:53:23.684767 5507 raft_consensus.cc:729] T e1ec5801ccef46058c0814de78ba3b79 P f040f18044544356a61ff6d7b60f15da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f040f18044544356a61ff6d7b60f15da, State: Initialized, Role: FOLLOWER
I20260929 17:53:23.684731 5509 raft_consensus.cc:348] T e1ec5801ccef46058c0814de78ba3b79 P ea3db7c9c7e849318988ddc7be9ab05c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea3db7c9c7e849318988ddc7be9ab05c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34223 } } peers { permanent_uuid: "f040f18044544356a61ff6d7b60f15da" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34325 } } peers { permanent_uuid: "7438304ccbf84631897c0b06fdfcfcc2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42171 } }
I20260929 17:53:23.684954 5509 raft_consensus.cc:374] T e1ec5801ccef46058c0814de78ba3b79 P ea3db7c9c7e849318988ddc7be9ab05c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:53:23.685015 5509 raft_consensus.cc:729] T e1ec5801ccef46058c0814de78ba3b79 P ea3db7c9c7e849318988ddc7be9ab05c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea3db7c9c7e849318988ddc7be9ab05c, State: Initialized, Role: FOLLOWER
I20260929 17:53:23.685196 5508 consensus_queue.cc:260] T e1ec5801ccef46058c0814de78ba3b79 P 7438304ccbf84631897c0b06fdfcfcc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea3db7c9c7e849318988ddc7be9ab05c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34223 } } peers { permanent_uuid: "f040f18044544356a61ff6d7b60f15da" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34325 } } peers { permanent_uuid: "7438304ccbf84631897c0b06fdfcfcc2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42171 } }
I20260929 17:53:23.685282 5507 consensus_queue.cc:260] T e1ec5801ccef46058c0814de78ba3b79 P f040f18044544356a61ff6d7b60f15da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea3db7c9c7e849318988ddc7be9ab05c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34223 } } peers { permanent_uuid: "f040f18044544356a61ff6d7b60f15da" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34325 } } peers { permanent_uuid: "7438304ccbf84631897c0b06fdfcfcc2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42171 } }
I20260929 17:53:23.685379 5509 consensus_queue.cc:260] T e1ec5801ccef46058c0814de78ba3b79 P ea3db7c9c7e849318988ddc7be9ab05c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea3db7c9c7e849318988ddc7be9ab05c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34223 } } peers { permanent_uuid: "f040f18044544356a61ff6d7b60f15da" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34325 } } peers { permanent_uuid: "7438304ccbf84631897c0b06fdfcfcc2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42171 } }
I20260929 17:53:23.685796 5490 heartbeater.cc:503] Master 127.0.98.62:40323 was elected leader, sending a full tablet report...
I20260929 17:53:23.686265 5509 ts_tablet_manager.cc:1428] T e1ec5801ccef46058c0814de78ba3b79 P ea3db7c9c7e849318988ddc7be9ab05c: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260929 17:53:23.686265 5507 ts_tablet_manager.cc:1428] T e1ec5801ccef46058c0814de78ba3b79 P f040f18044544356a61ff6d7b60f15da: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260929 17:53:23.686607 5228 heartbeater.cc:503] Master 127.0.98.62:40323 was elected leader, sending a full tablet report...
I20260929 17:53:23.687076 5508 ts_tablet_manager.cc:1428] T e1ec5801ccef46058c0814de78ba3b79 P 7438304ccbf84631897c0b06fdfcfcc2: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260929 17:53:23.687510 5359 heartbeater.cc:503] Master 127.0.98.62:40323 was elected leader, sending a full tablet report...
I20260929 17:53:23.701399 5513 raft_consensus.cc:482] T e1ec5801ccef46058c0814de78ba3b79 P 7438304ccbf84631897c0b06fdfcfcc2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 17:53:23.701574 5513 raft_consensus.cc:504] T e1ec5801ccef46058c0814de78ba3b79 P 7438304ccbf84631897c0b06fdfcfcc2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea3db7c9c7e849318988ddc7be9ab05c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34223 } } peers { permanent_uuid: "f040f18044544356a61ff6d7b60f15da" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34325 } } peers { permanent_uuid: "7438304ccbf84631897c0b06fdfcfcc2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42171 } }
I20260929 17:53:23.702594 5513 leader_election.cc:288] T e1ec5801ccef46058c0814de78ba3b79 P 7438304ccbf84631897c0b06fdfcfcc2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea3db7c9c7e849318988ddc7be9ab05c (127.0.98.1:34223), f040f18044544356a61ff6d7b60f15da (127.0.98.2:34325)
I20260929 17:53:23.704313 5492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:23.702815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34223 (local address 127.0.98.3:41221)
0929 17:53:23.702963 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:23.702978 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 17:53:23.703056 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:53:23.703326 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:53:23.703329 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:53:23.703344 (+ 15us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:53:23.703778 (+ 434us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 17:53:23.703780 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:53:23.703791 (+ 11us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 17:53:23.703830 (+ 39us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:53:23.704080 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:53:23.704083 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:53:23.704086 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 17:53:23.704157 (+ 71us) client_negotiation.cc:240] Negotiation successful
0929 17:53:23.704192 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":70}
I20260929 17:53:23.704332 5503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:23.702912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41221 (local address 127.0.98.1:34223)
0929 17:53:23.703027 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:23.703031 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:23.703044 (+ 13us) server_negotiation.cc:407] Connection header received
0929 17:53:23.703148 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:23.703151 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:23.703190 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:23.703295 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:53:23.703941 (+ 646us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:53:23.703945 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:53:23.703948 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 17:53:23.703960 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:53:23.703980 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 17:53:23.703990 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:53:23.704077 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0929 17:53:23.704157 (+ 80us) server_negotiation.cc:299] Negotiation successful
0929 17:53:23.704192 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260929 17:53:23.704468 5516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:23.702818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34325 (local address 127.0.98.3:35859)
0929 17:53:23.703289 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:23.703302 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 17:53:23.703391 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:53:23.703587 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:53:23.703590 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:53:23.703602 (+ 12us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:53:23.703962 (+ 360us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 17:53:23.703964 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:53:23.703974 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 17:53:23.703996 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:53:23.704216 (+ 220us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:53:23.704219 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:53:23.704222 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 17:53:23.704280 (+ 58us) client_negotiation.cc:240] Negotiation successful
0929 17:53:23.704311 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":416,"mutex_wait_us":169,"thread_start_us":103,"threads_started":1}
I20260929 17:53:23.704546 5506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:23.702945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35859 (local address 127.0.98.2:34325)
0929 17:53:23.703222 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:23.703226 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:23.703380 (+ 154us) server_negotiation.cc:407] Connection header received
0929 17:53:23.703466 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:23.703469 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:23.703501 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:23.703564 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:53:23.704095 (+ 531us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:53:23.704098 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:53:23.704101 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 17:53:23.704113 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:53:23.704135 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 17:53:23.704144 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:53:23.704249 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0929 17:53:23.704288 (+ 39us) server_negotiation.cc:299] Negotiation successful
0929 17:53:23.704321 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":199}
I20260929 17:53:23.704901 5182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1ec5801ccef46058c0814de78ba3b79" candidate_uuid: "7438304ccbf84631897c0b06fdfcfcc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea3db7c9c7e849318988ddc7be9ab05c" is_pre_election: true
I20260929 17:53:23.705066 5313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1ec5801ccef46058c0814de78ba3b79" candidate_uuid: "7438304ccbf84631897c0b06fdfcfcc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f040f18044544356a61ff6d7b60f15da" is_pre_election: true
I20260929 17:53:23.705253 5182 raft_consensus.cc:2454] T e1ec5801ccef46058c0814de78ba3b79 P ea3db7c9c7e849318988ddc7be9ab05c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7438304ccbf84631897c0b06fdfcfcc2 in term 0.
I20260929 17:53:23.705286 5313 raft_consensus.cc:2454] T e1ec5801ccef46058c0814de78ba3b79 P f040f18044544356a61ff6d7b60f15da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7438304ccbf84631897c0b06fdfcfcc2 in term 0.
I20260929 17:53:23.705698 5378 leader_election.cc:302] T e1ec5801ccef46058c0814de78ba3b79 P 7438304ccbf84631897c0b06fdfcfcc2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7438304ccbf84631897c0b06fdfcfcc2, ea3db7c9c7e849318988ddc7be9ab05c; no voters:
I20260929 17:53:23.705950 5513 raft_consensus.cc:2781] T e1ec5801ccef46058c0814de78ba3b79 P 7438304ccbf84631897c0b06fdfcfcc2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 17:53:23.706030 5513 raft_consensus.cc:482] T e1ec5801ccef46058c0814de78ba3b79 P 7438304ccbf84631897c0b06fdfcfcc2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 17:53:23.706074 5513 raft_consensus.cc:3037] T e1ec5801ccef46058c0814de78ba3b79 P 7438304ccbf84631897c0b06fdfcfcc2 [term 0 FOLLOWER]: Advancing to term 1
W20260929 17:53:23.706779 5360 tablet.cc:2369] T e1ec5801ccef46058c0814de78ba3b79 P f040f18044544356a61ff6d7b60f15da: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 17:53:23.707088 5513 raft_consensus.cc:504] T e1ec5801ccef46058c0814de78ba3b79 P 7438304ccbf84631897c0b06fdfcfcc2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea3db7c9c7e849318988ddc7be9ab05c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34223 } } peers { permanent_uuid: "f040f18044544356a61ff6d7b60f15da" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34325 } } peers { permanent_uuid: "7438304ccbf84631897c0b06fdfcfcc2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42171 } }
I20260929 17:53:23.707428 5513 leader_election.cc:288] T e1ec5801ccef46058c0814de78ba3b79 P 7438304ccbf84631897c0b06fdfcfcc2 [CANDIDATE]: Term 1 election: Requested vote from peers ea3db7c9c7e849318988ddc7be9ab05c (127.0.98.1:34223), f040f18044544356a61ff6d7b60f15da (127.0.98.2:34325)
I20260929 17:53:23.707758 5313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1ec5801ccef46058c0814de78ba3b79" candidate_uuid: "7438304ccbf84631897c0b06fdfcfcc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f040f18044544356a61ff6d7b60f15da"
I20260929 17:53:23.707758 5182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1ec5801ccef46058c0814de78ba3b79" candidate_uuid: "7438304ccbf84631897c0b06fdfcfcc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea3db7c9c7e849318988ddc7be9ab05c"
I20260929 17:53:23.707882 5313 raft_consensus.cc:3037] T e1ec5801ccef46058c0814de78ba3b79 P f040f18044544356a61ff6d7b60f15da [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:53:23.707883 5182 raft_consensus.cc:3037] T e1ec5801ccef46058c0814de78ba3b79 P ea3db7c9c7e849318988ddc7be9ab05c [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:53:23.709139 5182 raft_consensus.cc:2454] T e1ec5801ccef46058c0814de78ba3b79 P ea3db7c9c7e849318988ddc7be9ab05c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7438304ccbf84631897c0b06fdfcfcc2 in term 1.
I20260929 17:53:23.709138 5313 raft_consensus.cc:2454] T e1ec5801ccef46058c0814de78ba3b79 P f040f18044544356a61ff6d7b60f15da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7438304ccbf84631897c0b06fdfcfcc2 in term 1.
I20260929 17:53:23.709388 5378 leader_election.cc:302] T e1ec5801ccef46058c0814de78ba3b79 P 7438304ccbf84631897c0b06fdfcfcc2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7438304ccbf84631897c0b06fdfcfcc2, ea3db7c9c7e849318988ddc7be9ab05c; no voters:
I20260929 17:53:23.709535 5513 raft_consensus.cc:2781] T e1ec5801ccef46058c0814de78ba3b79 P 7438304ccbf84631897c0b06fdfcfcc2 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 17:53:23.709738 5513 raft_consensus.cc:686] T e1ec5801ccef46058c0814de78ba3b79 P 7438304ccbf84631897c0b06fdfcfcc2 [term 1 LEADER]: Becoming Leader. State: Replica: 7438304ccbf84631897c0b06fdfcfcc2, State: Running, Role: LEADER
I20260929 17:53:23.710036 5513 consensus_queue.cc:237] T e1ec5801ccef46058c0814de78ba3b79 P 7438304ccbf84631897c0b06fdfcfcc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea3db7c9c7e849318988ddc7be9ab05c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34223 } } peers { permanent_uuid: "f040f18044544356a61ff6d7b60f15da" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34325 } } peers { permanent_uuid: "7438304ccbf84631897c0b06fdfcfcc2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42171 } }
I20260929 17:53:23.713114 5040 catalog_manager.cc:5495] T e1ec5801ccef46058c0814de78ba3b79 P 7438304ccbf84631897c0b06fdfcfcc2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7438304ccbf84631897c0b06fdfcfcc2 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "7438304ccbf84631897c0b06fdfcfcc2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea3db7c9c7e849318988ddc7be9ab05c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34223 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f040f18044544356a61ff6d7b60f15da" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34325 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7438304ccbf84631897c0b06fdfcfcc2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42171 } health_report { overall_health: HEALTHY } } }
I20260929 17:53:23.727243 5505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:23.725895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46304 (local address 127.0.98.3:42171)
0929 17:53:23.726011 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:23.726015 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:23.726029 (+ 14us) server_negotiation.cc:407] Connection header received
0929 17:53:23.726137 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:23.726140 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:23.726198 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:23.726274 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:53:23.726884 (+ 610us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:53:23.726888 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:53:23.726890 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 17:53:23.726902 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:53:23.726920 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 17:53:23.726927 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:53:23.726999 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0929 17:53:23.727073 (+ 74us) server_negotiation.cc:299] Negotiation successful
0929 17:53:23.727111 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260929 17:53:23.732872 5182 raft_consensus.cc:1261] T e1ec5801ccef46058c0814de78ba3b79 P ea3db7c9c7e849318988ddc7be9ab05c [term 1 FOLLOWER]: Refusing update from remote peer 7438304ccbf84631897c0b06fdfcfcc2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 17:53:23.732870 5313 raft_consensus.cc:1261] T e1ec5801ccef46058c0814de78ba3b79 P f040f18044544356a61ff6d7b60f15da [term 1 FOLLOWER]: Refusing update from remote peer 7438304ccbf84631897c0b06fdfcfcc2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 17:53:23.733597 5517 consensus_queue.cc:1035] T e1ec5801ccef46058c0814de78ba3b79 P 7438304ccbf84631897c0b06fdfcfcc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea3db7c9c7e849318988ddc7be9ab05c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 17:53:23.733786 5513 consensus_queue.cc:1035] T e1ec5801ccef46058c0814de78ba3b79 P 7438304ccbf84631897c0b06fdfcfcc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f040f18044544356a61ff6d7b60f15da" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 17:53:23.742659 5521 mvcc.cc:205] Tried to move back new op lower bound from 7334725237683150848 to 7334725237598896128. Current Snapshot: MvccSnapshot[applied={T|T < 7334725237683150848 or (T in {7334725237683150848})}]
I20260929 17:53:23.751530 5039 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56930:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 17:53:23.751720 5039 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56930:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 17:53:23.754180 5039 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8b46676206024b629b69b10115928a41: Sending DeleteTablet for 3 replicas of tablet e1ec5801ccef46058c0814de78ba3b79
I20260929 17:53:23.754940 5293 tablet_service.cc:1514] Processing DeleteTablet for tablet e1ec5801ccef46058c0814de78ba3b79 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 17:53:23 UTC) from {username='slave'} at 127.0.0.1:48140
I20260929 17:53:23.754957 5424 tablet_service.cc:1514] Processing DeleteTablet for tablet e1ec5801ccef46058c0814de78ba3b79 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 17:53:23 UTC) from {username='slave'} at 127.0.0.1:46298
I20260929 17:53:23.754954 5162 tablet_service.cc:1514] Processing DeleteTablet for tablet e1ec5801ccef46058c0814de78ba3b79 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 17:53:23 UTC) from {username='slave'} at 127.0.0.1:40430
I20260929 17:53:23.755430 5532 tablet_replica.cc:330] T e1ec5801ccef46058c0814de78ba3b79 P 7438304ccbf84631897c0b06fdfcfcc2: stopping tablet replica
I20260929 17:53:23.755551 5530 tablet_replica.cc:330] T e1ec5801ccef46058c0814de78ba3b79 P f040f18044544356a61ff6d7b60f15da: stopping tablet replica
I20260929 17:53:23.755767 5532 raft_consensus.cc:2229] T e1ec5801ccef46058c0814de78ba3b79 P 7438304ccbf84631897c0b06fdfcfcc2 [term 1 LEADER]: Raft consensus shutting down.
I20260929 17:53:23.755837 5530 raft_consensus.cc:2229] T e1ec5801ccef46058c0814de78ba3b79 P f040f18044544356a61ff6d7b60f15da [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 17:53:23.756047 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskup3Soe/build/debug/bin/kudu with pid 5100
I20260929 17:53:23.756171 5530 raft_consensus.cc:2258] T e1ec5801ccef46058c0814de78ba3b79 P f040f18044544356a61ff6d7b60f15da [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 17:53:23.756175 5532 raft_consensus.cc:2258] T e1ec5801ccef46058c0814de78ba3b79 P 7438304ccbf84631897c0b06fdfcfcc2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 17:53:23.757869 5530 ts_tablet_manager.cc:1905] T e1ec5801ccef46058c0814de78ba3b79 P f040f18044544356a61ff6d7b60f15da: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 17:53:23.757869 5532 ts_tablet_manager.cc:1905] T e1ec5801ccef46058c0814de78ba3b79 P 7438304ccbf84631897c0b06fdfcfcc2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 17:53:23.760433 5532 ts_tablet_manager.cc:1918] T e1ec5801ccef46058c0814de78ba3b79 P 7438304ccbf84631897c0b06fdfcfcc2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 17:53:23.760422 5530 ts_tablet_manager.cc:1918] T e1ec5801ccef46058c0814de78ba3b79 P f040f18044544356a61ff6d7b60f15da: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 17:53:23.760516 5530 log.cc:1196] T e1ec5801ccef46058c0814de78ba3b79 P f040f18044544356a61ff6d7b60f15da: Deleting WAL directory at /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790704375975685-392-0/minicluster-data/ts-1/wal/wals/e1ec5801ccef46058c0814de78ba3b79
I20260929 17:53:23.760588 5532 log.cc:1196] T e1ec5801ccef46058c0814de78ba3b79 P 7438304ccbf84631897c0b06fdfcfcc2: Deleting WAL directory at /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790704375975685-392-0/minicluster-data/ts-2/wal/wals/e1ec5801ccef46058c0814de78ba3b79
I20260929 17:53:23.760872 5530 ts_tablet_manager.cc:1939] T e1ec5801ccef46058c0814de78ba3b79 P f040f18044544356a61ff6d7b60f15da: Deleting consensus metadata
I20260929 17:53:23.760929 5532 ts_tablet_manager.cc:1939] T e1ec5801ccef46058c0814de78ba3b79 P 7438304ccbf84631897c0b06fdfcfcc2: Deleting consensus metadata
I20260929 17:53:23.762060 5026 catalog_manager.cc:4841] TS f040f18044544356a61ff6d7b60f15da (127.0.98.2:34325): tablet e1ec5801ccef46058c0814de78ba3b79 (table test-table [id=9200e80e936e4de98b5f1dee3fe4ff29]) successfully deleted
I20260929 17:53:23.762328 5027 catalog_manager.cc:4841] TS 7438304ccbf84631897c0b06fdfcfcc2 (127.0.98.3:42171): tablet e1ec5801ccef46058c0814de78ba3b79 (table test-table [id=9200e80e936e4de98b5f1dee3fe4ff29]) successfully deleted
W20260929 17:53:23.765255 5025 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:34223 (error 108)
I20260929 17:53:23.766023 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskup3Soe/build/debug/bin/kudu with pid 5231
I20260929 17:53:23.766201 5501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:23.765922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34223 (local address 127.0.0.1:40434)
0929 17:53:23.766058 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:23.766123 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260929 17:53:23.768144 5025 catalog_manager.cc:4584] TS ea3db7c9c7e849318988ddc7be9ab05c (127.0.98.1:34223): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e1ec5801ccef46058c0814de78ba3b79: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34223: connect: Connection refused (error 111)
I20260929 17:53:23.770956 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskup3Soe/build/debug/bin/kudu with pid 5362
I20260929 17:53:23.775655 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskup3Soe/build/debug/bin/kudu with pid 5009
2026-09-29T17:53:23Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (897 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260929 17:53:23.795104 392 mini_oidc.cc:152] Starting JWKS server
I20260929 17:53:23.796613 392 webserver.cc:460] Webserver started at https://127.0.0.1:46389/ using document root <none> and password file <none>
I20260929 17:53:23.796765 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260929 17:53:23.797031 392 webserver.cc:460] Webserver started at http://127.0.0.1:40427/ using document root <none> and password file <none>
2026-09-29T17:53:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T17:53:23Z Disabled control of system clock
I20260929 17:53:23.804726 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskup3Soe/build/debug/bin/kudu
/tmp/dist-test-taskup3Soe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790704375975685-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790704375975685-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790704375975685-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790704375975685-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38749
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:38253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38749
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46389/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790704375975685-392-0/testchainca.cert with env {}
W20260929 17:53:23.911331 5539 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:53:23.911603 5539 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:53:23.911659 5539 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:53:23.915758 5539 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 17:53:23.915817 5539 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:53:23.915840 5539 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:53:23.915861 5539 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 17:53:23.915889 5539 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260929 17:53:23.915918 5539 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260929 17:53:23.915944 5539 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:46389/jwks.json
I20260929 17:53:23.921222 5539 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790704375975685-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790704375975685-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38749
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38749
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46389/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790704375975685-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790704375975685-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5539
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790704375975685-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:53:23.922255 5539 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:53:23.923230 5539 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:53:23.931309 5545 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:53:23.931316 5544 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:53:23.931366 5539 server_base.cc:1034] running on GCE node
W20260929 17:53:23.931325 5547 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:53:23.932034 5539 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:53:23.933192 5539 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:53:23.934376 5539 hybrid_clock.cc:648] HybridClock initialized: now 1790704403934362 us; error 27 us; skew 500 ppm
I20260929 17:53:23.936127 5539 webserver.cc:460] Webserver started at http://127.0.98.62:34267/ using document root <none> and password file <none>
I20260929 17:53:23.936690 5539 fs_manager.cc:359] Metadata directory not provided
I20260929 17:53:23.936755 5539 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:53:23.936935 5539 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:53:23.938999 5539 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790704375975685-392-0/minicluster-data/master-0/data/instance:
uuid: "7aabf468beac4e2d86901994ea9baedc"
format_stamp: "Formatted at 2026-09-29 17:53:23 on dist-test-slave-53nh"
server_key: "eded381e2ef1c50ce4a0026dd275d437"
server_key_iv: "caf24d197e97deb80d85c0c6bef800ca"
server_key_version: "encryptionkey@0"
I20260929 17:53:23.939559 5539 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790704375975685-392-0/minicluster-data/master-0/wal/instance:
uuid: "7aabf468beac4e2d86901994ea9baedc"
format_stamp: "Formatted at 2026-09-29 17:53:23 on dist-test-slave-53nh"
server_key: "eded381e2ef1c50ce4a0026dd275d437"
server_key_iv: "caf24d197e97deb80d85c0c6bef800ca"
server_key_version: "encryptionkey@0"
I20260929 17:53:23.943197 5539 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260929 17:53:23.945776 5553 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:53:23.947005 5539 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 17:53:23.947132 5539 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790704375975685-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790704375975685-392-0/minicluster-data/master-0/wal
uuid: "7aabf468beac4e2d86901994ea9baedc"
format_stamp: "Formatted at 2026-09-29 17:53:23 on dist-test-slave-53nh"
server_key: "eded381e2ef1c50ce4a0026dd275d437"
server_key_iv: "caf24d197e97deb80d85c0c6bef800ca"
server_key_version: "encryptionkey@0"
I20260929 17:53:23.947234 5539 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790704375975685-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790704375975685-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790704375975685-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:53:23.958772 5539 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:53:23.959463 5539 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:53:23.959636 5539 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:53:23.966127 5539 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38749
I20260929 17:53:23.966136 5605 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38749 every 8 connection(s)
I20260929 17:53:23.967067 5539 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790704375975685-392-0/minicluster-data/master-0/data/info.pb
I20260929 17:53:23.969640 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskup3Soe/build/debug/bin/kudu as pid 5539
I20260929 17:53:23.969756 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790704375975685-392-0/minicluster-data/master-0/wal/instance
I20260929 17:53:23.970057 392 external_mini_cluster.cc:1428] Setting key c7c7123404dbef26ce8a2847f85ffe1d
I20260929 17:53:23.970479 5606 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:53:23.976962 5606 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7aabf468beac4e2d86901994ea9baedc: Bootstrap starting.
I20260929 17:53:23.979470 5609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:23.971412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50702 (local address 127.0.98.62:38749)
0929 17:53:23.972042 (+ 630us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:23.972057 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:23.972094 (+ 37us) server_negotiation.cc:407] Connection header received
0929 17:53:23.973088 (+ 994us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:23.973107 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:23.973514 (+ 407us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:23.973829 (+ 315us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:53:23.974246 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:23.975205 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:23.976086 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:23.976569 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:23.976717 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:23.977677 (+ 960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:53:23.977682 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:53:23.977696 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 17:53:23.978114 (+ 418us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:53:23.978172 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 17:53:23.978192 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:53:23.978278 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0929 17:53:23.978448 (+ 170us) server_negotiation.cc:299] Negotiation successful
0929 17:53:23.978717 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":131,"threads_started":1}
I20260929 17:53:23.979485 5606 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7aabf468beac4e2d86901994ea9baedc: Neither blocks nor log segments found. Creating new log.
I20260929 17:53:23.980307 5606 log.cc:824] T 00000000000000000000000000000000 P 7aabf468beac4e2d86901994ea9baedc: Log is configured to *not* fsync() on all Append() calls
I20260929 17:53:23.982564 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 17:53:23.982666 5606 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7aabf468beac4e2d86901994ea9baedc: No bootstrap required, opened a new log
I20260929 17:53:23.986447 5606 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7aabf468beac4e2d86901994ea9baedc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7aabf468beac4e2d86901994ea9baedc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38749 } }
I20260929 17:53:23.986727 5606 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7aabf468beac4e2d86901994ea9baedc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:53:23.986792 5606 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7aabf468beac4e2d86901994ea9baedc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7aabf468beac4e2d86901994ea9baedc, State: Initialized, Role: FOLLOWER
I20260929 17:53:23.987249 5606 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7aabf468beac4e2d86901994ea9baedc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7aabf468beac4e2d86901994ea9baedc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38749 } }
I20260929 17:53:23.987398 5606 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7aabf468beac4e2d86901994ea9baedc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 17:53:23.987470 5606 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7aabf468beac4e2d86901994ea9baedc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 17:53:23.987586 5606 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7aabf468beac4e2d86901994ea9baedc [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:53:23.988694 5606 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7aabf468beac4e2d86901994ea9baedc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7aabf468beac4e2d86901994ea9baedc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38749 } }
I20260929 17:53:23.989086 5606 leader_election.cc:302] T 00000000000000000000000000000000 P 7aabf468beac4e2d86901994ea9baedc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7aabf468beac4e2d86901994ea9baedc; no voters:
I20260929 17:53:23.989419 5606 leader_election.cc:288] T 00000000000000000000000000000000 P 7aabf468beac4e2d86901994ea9baedc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 17:53:23.989547 5611 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7aabf468beac4e2d86901994ea9baedc [term 1 FOLLOWER]: Leader election won for term 1
I20260929 17:53:23.989773 5611 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7aabf468beac4e2d86901994ea9baedc [term 1 LEADER]: Becoming Leader. State: Replica: 7aabf468beac4e2d86901994ea9baedc, State: Running, Role: LEADER
I20260929 17:53:23.990162 5611 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7aabf468beac4e2d86901994ea9baedc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7aabf468beac4e2d86901994ea9baedc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38749 } }
I20260929 17:53:23.990736 5606 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7aabf468beac4e2d86901994ea9baedc [sys.catalog]: configured and running, proceeding with master startup.
I20260929 17:53:23.991844 5612 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7aabf468beac4e2d86901994ea9baedc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7aabf468beac4e2d86901994ea9baedc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7aabf468beac4e2d86901994ea9baedc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38749 } } }
I20260929 17:53:23.991842 5613 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7aabf468beac4e2d86901994ea9baedc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7aabf468beac4e2d86901994ea9baedc. Latest consensus state: current_term: 1 leader_uuid: "7aabf468beac4e2d86901994ea9baedc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7aabf468beac4e2d86901994ea9baedc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38749 } } }
I20260929 17:53:23.991994 5612 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7aabf468beac4e2d86901994ea9baedc [sys.catalog]: This master's current role is: LEADER
I20260929 17:53:23.992007 5613 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7aabf468beac4e2d86901994ea9baedc [sys.catalog]: This master's current role is: LEADER
I20260929 17:53:23.993158 5620 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 17:53:23.995968 5620 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 17:53:24.002434 5620 catalog_manager.cc:1347] Generated new cluster ID: 5cea876ceec54ccf9bbbbc2334932ad5
I20260929 17:53:24.002511 5620 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 17:53:24.009249 5620 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 17:53:24.010186 5620 catalog_manager.cc:1504] Loading token signing keys...
I20260929 17:53:24.017530 5620 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7aabf468beac4e2d86901994ea9baedc: Generated new TSK 0
I20260929 17:53:24.018220 5620 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260929 17:53:24.050414 5636 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260929 17:53:24.050544 5636 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260929 17:53:24.051113 5609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:24.034000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50706 (local address 127.0.98.62:38749)
0929 17:53:24.034129 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:24.034134 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:24.034302 (+ 168us) server_negotiation.cc:407] Connection header received
0929 17:53:24.034505 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:24.034509 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:24.034568 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:24.034688 (+ 120us) server_negotiation.cc:226] Negotiated authn=JWT
0929 17:53:24.034933 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:24.035749 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:24.036587 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:24.036917 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:24.037045 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:24.037403 (+ 358us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0929 17:53:24.050704 (+ 13301us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46389/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0929 17:53:24.050972 (+ 268us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:50706: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46389/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":46}
W20260929 17:53:24.051179 5609 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:50706: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46389/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260929 17:53:24.052520 5635 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:53:24.033879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38749 (local address 127.0.0.1:50706)
0929 17:53:24.034225 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:24.034238 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 17:53:24.034351 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:53:24.034721 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:53:24.034724 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:53:24.034731 (+ 7us) client_negotiation.cc:189] Negotiated authn=JWT
0929 17:53:24.034841 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:24.034846 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:24.035854 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:24.035856 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:24.036474 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:24.036480 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:24.037077 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:24.037095 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:24.037172 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:24.037220 (+ 48us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0929 17:53:24.050943 (+ 13723us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46389/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0929 17:53:24.051174 (+ 231us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:38749: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46389/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":117,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.0.98.62:38749: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46389/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260929 17:53:24.053397 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskup3Soe/build/debug/bin/kudu with pid 5539
2026-09-29T17:53:24Z chronyd exiting
I20260929 17:53:24.089643 392 test_util.cc:175] -----------------------------------------------
I20260929 17:53:24.089735 392 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790704375975685-392-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (295 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260929 17:53:24.090756 392 mini_oidc.cc:152] Starting JWKS server
I20260929 17:53:24.091557 392 webserver.cc:460] Webserver started at https://127.0.0.1:40927/ using document root <none> and password file <none>
I20260929 17:53:24.091637 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260929 17:53:24.091845 392 webserver.cc:460] Webserver started at http://127.0.0.1:39197/ using document root <none> and password file <none>
2026-09-29T17:53:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T17:53:24Z Disabled control of system clock
I20260929 17:53:24.099671 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskup3Soe/build/debug/bin/kudu
/tmp/dist-test-taskup3Soe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790704375975685-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790704375975685-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790704375975685-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790704375975685-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44765
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:35763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44765
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40927/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790704375975685-392-0/testchainca.cert with env {}
W20260929 17:53:24.205947 5642 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:53:24.206233 5642 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:53:24.206291 5642 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:53:24.210282 5642 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 17:53:24.210350 5642 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:53:24.210371 5642 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:53:24.210390 5642 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 17:53:24.210412 5642 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260929 17:53:24.210438 5642 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260929 17:53:24.210464 5642 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:40927/jwks.json
I20260929 17:53:24.215771 5642 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790704375975685-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790704375975685-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44765
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44765
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40927/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790704375975685-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790704375975685-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5642
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790704375975685-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:53:24.216883 5642 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:53:24.218101 5642 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:53:24.226424 5647 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:53:24.226449 5650 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:53:24.226449 5648 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:53:24.226588 5642 server_base.cc:1034] running on GCE node
I20260929 17:53:24.227216 5642 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:53:24.228374 5642 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:53:24.229535 5642 hybrid_clock.cc:648] HybridClock initialized: now 1790704404229519 us; error 30 us; skew 500 ppm
I20260929 17:53:24.231395 5642 webserver.cc:460] Webserver started at http://127.0.98.62:35093/ using document root <none> and password file <none>
I20260929 17:53:24.231935 5642 fs_manager.cc:359] Metadata directory not provided
I20260929 17:53:24.231992 5642 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:53:24.232151 5642 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:53:24.234234 5642 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790704375975685-392-0/minicluster-data/master-0/data/instance:
uuid: "4d11e06387c94fbcb50ab78c0d30cf57"
format_stamp: "Formatted at 2026-09-29 17:53:24 on dist-test-slave-53nh"
server_key: "1b78f91dd9c4f4266c652fbbbe6808dd"
server_key_iv: "45f0ee921682bc7fa218a0d95c0ddb0d"
server_key_version: "encryptionkey@0"
I20260929 17:53:24.234756 5642 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790704375975685-392-0/minicluster-data/master-0/wal/instance:
uuid: "4d11e06387c94fbcb50ab78c0d30cf57"
format_stamp: "Formatted at 2026-09-29 17:53:24 on dist-test-slave-53nh"
server_key: "1b78f91dd9c4f4266c652fbbbe6808dd"
server_key_iv: "45f0ee921682bc7fa218a0d95c0ddb0d"
server_key_version: "encryptionkey@0"
I20260929 17:53:24.238471 5642 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 17:53:24.241112 5656 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:53:24.242556 5642 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 17:53:24.242681 5642 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790704375975685-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790704375975685-392-0/minicluster-data/master-0/wal
uuid: "4d11e06387c94fbcb50ab78c0d30cf57"
format_stamp: "Formatted at 2026-09-29 17:53:24 on dist-test-slave-53nh"
server_key: "1b78f91dd9c4f4266c652fbbbe6808dd"
server_key_iv: "45f0ee921682bc7fa218a0d95c0ddb0d"
server_key_version: "encryptionkey@0"
I20260929 17:53:24.242807 5642 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790704375975685-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790704375975685-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790704375975685-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:53:24.260710 5642 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:53:24.261503 5642 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:53:24.261679 5642 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:53:24.268539 5642 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44765
I20260929 17:53:24.268590 5708 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44765 every 8 connection(s)
I20260929 17:53:24.269534 5642 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790704375975685-392-0/minicluster-data/master-0/data/info.pb
I20260929 17:53:24.272663 5709 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:53:24.274966 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskup3Soe/build/debug/bin/kudu as pid 5642
I20260929 17:53:24.275064 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790704375975685-392-0/minicluster-data/master-0/wal/instance
I20260929 17:53:24.275300 392 external_mini_cluster.cc:1428] Setting key 3152d337f3eede0c464f0591944222f7
I20260929 17:53:24.278566 5709 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d11e06387c94fbcb50ab78c0d30cf57: Bootstrap starting.
I20260929 17:53:24.280725 5709 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4d11e06387c94fbcb50ab78c0d30cf57: Neither blocks nor log segments found. Creating new log.
I20260929 17:53:24.281445 5709 log.cc:824] T 00000000000000000000000000000000 P 4d11e06387c94fbcb50ab78c0d30cf57: Log is configured to *not* fsync() on all Append() calls
I20260929 17:53:24.283526 5709 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d11e06387c94fbcb50ab78c0d30cf57: No bootstrap required, opened a new log
I20260929 17:53:24.284284 5712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:24.276670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41378 (local address 127.0.98.62:44765)
0929 17:53:24.277350 (+ 680us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:24.277366 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:24.277404 (+ 38us) server_negotiation.cc:407] Connection header received
0929 17:53:24.278036 (+ 632us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:24.278055 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:24.278479 (+ 424us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:24.278794 (+ 315us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:53:24.279237 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:24.280202 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:24.281144 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:24.281644 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:24.281790 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:24.282574 (+ 784us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:53:24.282579 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:53:24.282592 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 17:53:24.282983 (+ 391us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:53:24.283041 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 17:53:24.283061 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:53:24.283151 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0929 17:53:24.283286 (+ 135us) server_negotiation.cc:299] Negotiation successful
0929 17:53:24.283550 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282,"thread_start_us":128,"threads_started":1}
I20260929 17:53:24.287176 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 17:53:24.287420 5709 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4d11e06387c94fbcb50ab78c0d30cf57 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d11e06387c94fbcb50ab78c0d30cf57" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44765 } }
I20260929 17:53:24.287649 5709 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4d11e06387c94fbcb50ab78c0d30cf57 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:53:24.287704 5709 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4d11e06387c94fbcb50ab78c0d30cf57 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d11e06387c94fbcb50ab78c0d30cf57, State: Initialized, Role: FOLLOWER
I20260929 17:53:24.288110 5709 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4d11e06387c94fbcb50ab78c0d30cf57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d11e06387c94fbcb50ab78c0d30cf57" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44765 } }
I20260929 17:53:24.288261 5709 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4d11e06387c94fbcb50ab78c0d30cf57 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 17:53:24.288324 5709 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4d11e06387c94fbcb50ab78c0d30cf57 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 17:53:24.288411 5709 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4d11e06387c94fbcb50ab78c0d30cf57 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:53:24.289361 5709 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4d11e06387c94fbcb50ab78c0d30cf57 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d11e06387c94fbcb50ab78c0d30cf57" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44765 } }
I20260929 17:53:24.289660 5709 leader_election.cc:302] T 00000000000000000000000000000000 P 4d11e06387c94fbcb50ab78c0d30cf57 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4d11e06387c94fbcb50ab78c0d30cf57; no voters:
I20260929 17:53:24.290107 5709 leader_election.cc:288] T 00000000000000000000000000000000 P 4d11e06387c94fbcb50ab78c0d30cf57 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 17:53:24.290199 5714 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4d11e06387c94fbcb50ab78c0d30cf57 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 17:53:24.290413 5714 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4d11e06387c94fbcb50ab78c0d30cf57 [term 1 LEADER]: Becoming Leader. State: Replica: 4d11e06387c94fbcb50ab78c0d30cf57, State: Running, Role: LEADER
I20260929 17:53:24.290750 5714 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4d11e06387c94fbcb50ab78c0d30cf57 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d11e06387c94fbcb50ab78c0d30cf57" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44765 } }
I20260929 17:53:24.291476 5709 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4d11e06387c94fbcb50ab78c0d30cf57 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 17:53:24.292279 5715 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d11e06387c94fbcb50ab78c0d30cf57 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4d11e06387c94fbcb50ab78c0d30cf57. Latest consensus state: current_term: 1 leader_uuid: "4d11e06387c94fbcb50ab78c0d30cf57" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d11e06387c94fbcb50ab78c0d30cf57" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44765 } } }
I20260929 17:53:24.292423 5715 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d11e06387c94fbcb50ab78c0d30cf57 [sys.catalog]: This master's current role is: LEADER
I20260929 17:53:24.292410 5716 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d11e06387c94fbcb50ab78c0d30cf57 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4d11e06387c94fbcb50ab78c0d30cf57" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d11e06387c94fbcb50ab78c0d30cf57" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44765 } } }
I20260929 17:53:24.292721 5716 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d11e06387c94fbcb50ab78c0d30cf57 [sys.catalog]: This master's current role is: LEADER
I20260929 17:53:24.293763 5723 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 17:53:24.296265 5723 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 17:53:24.301304 5723 catalog_manager.cc:1347] Generated new cluster ID: 35aa35b79c094b568fda0c1b6a816926
I20260929 17:53:24.301364 5723 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 17:53:24.316987 5723 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 17:53:24.317960 5723 catalog_manager.cc:1504] Loading token signing keys...
I20260929 17:53:24.331358 5723 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4d11e06387c94fbcb50ab78c0d30cf57: Generated new TSK 0
I20260929 17:53:24.332242 5723 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260929 17:53:24.365694 5739 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260929 17:53:24.365782 5739 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260929 17:53:24.366581 5712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:24.349970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41384 (local address 127.0.98.62:44765)
0929 17:53:24.350099 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:24.350104 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:24.350301 (+ 197us) server_negotiation.cc:407] Connection header received
0929 17:53:24.350445 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:24.350448 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:24.350503 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:24.350621 (+ 118us) server_negotiation.cc:226] Negotiated authn=JWT
0929 17:53:24.350862 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:24.352202 (+ 1340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:24.353105 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:24.353467 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:24.353569 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:24.353852 (+ 283us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0929 17:53:24.366112 (+ 12260us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40927/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0929 17:53:24.366436 (+ 324us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:41384: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40927/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":47}
W20260929 17:53:24.366653 5712 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:41384: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40927/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260929 17:53:24.366652 5738 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:53:24.349831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44765 (local address 127.0.0.1:41384)
0929 17:53:24.350224 (+ 393us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:24.350236 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 17:53:24.350330 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:53:24.350645 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:53:24.350648 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:53:24.350655 (+ 7us) client_negotiation.cc:189] Negotiated authn=JWT
0929 17:53:24.350758 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:24.350763 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:24.352354 (+ 1591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:24.352357 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:24.352981 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:24.352988 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:24.353567 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:24.353585 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:24.353663 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:24.353688 (+ 25us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0929 17:53:24.366352 (+ 12664us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40927/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0929 17:53:24.366507 (+ 155us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:44765: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40927/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":129,"threads_started":1}
I20260929 17:53:24.367379 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskup3Soe/build/debug/bin/kudu with pid 5642
2026-09-29T17:53:24Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (314 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260929 17:53:24.405781 392 mini_oidc.cc:152] Starting JWKS server
I20260929 17:53:24.406638 392 webserver.cc:460] Webserver started at https://127.0.0.1:43295/ using document root <none> and password file <none>
I20260929 17:53:24.406729 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260929 17:53:24.406991 392 webserver.cc:460] Webserver started at http://127.0.0.1:37459/ using document root <none> and password file <none>
2026-09-29T17:53:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T17:53:24Z Disabled control of system clock
I20260929 17:53:24.415750 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskup3Soe/build/debug/bin/kudu
/tmp/dist-test-taskup3Soe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790704375975685-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790704375975685-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790704375975685-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790704375975685-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40979
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:45097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40979
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43295/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260929 17:53:24.525801 5745 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:53:24.526067 5745 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:53:24.526104 5745 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:53:24.530131 5745 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 17:53:24.530218 5745 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:53:24.530239 5745 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:53:24.530261 5745 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 17:53:24.530282 5745 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260929 17:53:24.530313 5745 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260929 17:53:24.530339 5745 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:43295/jwks.json
I20260929 17:53:24.535089 5745 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790704375975685-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790704375975685-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40979
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40979
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43295/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790704375975685-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5745
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790704375975685-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:53:24.536127 5745 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:53:24.537540 5745 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:53:24.545919 5753 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:53:24.545954 5751 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:53:24.546017 5745 server_base.cc:1034] running on GCE node
W20260929 17:53:24.545974 5750 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:53:24.546777 5745 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:53:24.547873 5745 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:53:24.549055 5745 hybrid_clock.cc:648] HybridClock initialized: now 1790704404549032 us; error 31 us; skew 500 ppm
I20260929 17:53:24.550984 5745 webserver.cc:460] Webserver started at http://127.0.98.62:38167/ using document root <none> and password file <none>
I20260929 17:53:24.551568 5745 fs_manager.cc:359] Metadata directory not provided
I20260929 17:53:24.551637 5745 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:53:24.551851 5745 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:53:24.554024 5745 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790704375975685-392-0/minicluster-data/master-0/data/instance:
uuid: "cefc59b182204f3f8862506f568f754d"
format_stamp: "Formatted at 2026-09-29 17:53:24 on dist-test-slave-53nh"
server_key: "9d7e221402d67f86213455d57638c1f5"
server_key_iv: "f22df4f5583e58acbb76a8768cc10647"
server_key_version: "encryptionkey@0"
I20260929 17:53:24.554672 5745 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790704375975685-392-0/minicluster-data/master-0/wal/instance:
uuid: "cefc59b182204f3f8862506f568f754d"
format_stamp: "Formatted at 2026-09-29 17:53:24 on dist-test-slave-53nh"
server_key: "9d7e221402d67f86213455d57638c1f5"
server_key_iv: "f22df4f5583e58acbb76a8768cc10647"
server_key_version: "encryptionkey@0"
I20260929 17:53:24.558732 5745 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 17:53:24.561395 5759 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:53:24.562677 5745 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 17:53:24.562803 5745 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790704375975685-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790704375975685-392-0/minicluster-data/master-0/wal
uuid: "cefc59b182204f3f8862506f568f754d"
format_stamp: "Formatted at 2026-09-29 17:53:24 on dist-test-slave-53nh"
server_key: "9d7e221402d67f86213455d57638c1f5"
server_key_iv: "f22df4f5583e58acbb76a8768cc10647"
server_key_version: "encryptionkey@0"
I20260929 17:53:24.562919 5745 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790704375975685-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790704375975685-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790704375975685-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:53:24.580925 5745 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:53:24.581740 5745 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:53:24.581943 5745 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:53:24.588848 5745 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40979
I20260929 17:53:24.588865 5811 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40979 every 8 connection(s)
I20260929 17:53:24.589902 5745 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790704375975685-392-0/minicluster-data/master-0/data/info.pb
I20260929 17:53:24.591359 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskup3Soe/build/debug/bin/kudu as pid 5745
I20260929 17:53:24.591462 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790704375975685-392-0/minicluster-data/master-0/wal/instance
I20260929 17:53:24.591737 392 external_mini_cluster.cc:1428] Setting key b754083e28fc55ac0b1e7fff5c12ebdf
I20260929 17:53:24.593483 5812 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:53:24.599071 5812 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cefc59b182204f3f8862506f568f754d: Bootstrap starting.
I20260929 17:53:24.600754 5815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:24.593063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38624 (local address 127.0.98.62:40979)
0929 17:53:24.593808 (+ 745us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:24.593828 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:24.593865 (+ 37us) server_negotiation.cc:407] Connection header received
0929 17:53:24.594522 (+ 657us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:24.594539 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:24.594948 (+ 409us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:24.595292 (+ 344us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:53:24.595723 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:24.596690 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:24.597620 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:24.598080 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:24.598223 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:24.599066 (+ 843us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:53:24.599072 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:53:24.599088 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 17:53:24.599500 (+ 412us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:53:24.599555 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 17:53:24.599585 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:53:24.599667 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0929 17:53:24.599800 (+ 133us) server_negotiation.cc:299] Negotiation successful
0929 17:53:24.600047 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":314,"thread_start_us":144,"threads_started":1}
I20260929 17:53:24.601439 5812 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cefc59b182204f3f8862506f568f754d: Neither blocks nor log segments found. Creating new log.
I20260929 17:53:24.602278 5812 log.cc:824] T 00000000000000000000000000000000 P cefc59b182204f3f8862506f568f754d: Log is configured to *not* fsync() on all Append() calls
I20260929 17:53:24.603685 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 17:53:24.604384 5812 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cefc59b182204f3f8862506f568f754d: No bootstrap required, opened a new log
I20260929 17:53:24.608210 5812 raft_consensus.cc:348] T 00000000000000000000000000000000 P cefc59b182204f3f8862506f568f754d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cefc59b182204f3f8862506f568f754d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40979 } }
I20260929 17:53:24.608477 5812 raft_consensus.cc:374] T 00000000000000000000000000000000 P cefc59b182204f3f8862506f568f754d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:53:24.608522 5812 raft_consensus.cc:729] T 00000000000000000000000000000000 P cefc59b182204f3f8862506f568f754d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cefc59b182204f3f8862506f568f754d, State: Initialized, Role: FOLLOWER
I20260929 17:53:24.608896 5812 consensus_queue.cc:260] T 00000000000000000000000000000000 P cefc59b182204f3f8862506f568f754d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cefc59b182204f3f8862506f568f754d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40979 } }
I20260929 17:53:24.609037 5812 raft_consensus.cc:388] T 00000000000000000000000000000000 P cefc59b182204f3f8862506f568f754d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 17:53:24.609102 5812 raft_consensus.cc:482] T 00000000000000000000000000000000 P cefc59b182204f3f8862506f568f754d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 17:53:24.609174 5812 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cefc59b182204f3f8862506f568f754d [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:53:24.610020 5812 raft_consensus.cc:504] T 00000000000000000000000000000000 P cefc59b182204f3f8862506f568f754d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cefc59b182204f3f8862506f568f754d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40979 } }
I20260929 17:53:24.610307 5812 leader_election.cc:302] T 00000000000000000000000000000000 P cefc59b182204f3f8862506f568f754d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cefc59b182204f3f8862506f568f754d; no voters:
I20260929 17:53:24.610575 5812 leader_election.cc:288] T 00000000000000000000000000000000 P cefc59b182204f3f8862506f568f754d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 17:53:24.610685 5817 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cefc59b182204f3f8862506f568f754d [term 1 FOLLOWER]: Leader election won for term 1
I20260929 17:53:24.610896 5817 raft_consensus.cc:686] T 00000000000000000000000000000000 P cefc59b182204f3f8862506f568f754d [term 1 LEADER]: Becoming Leader. State: Replica: cefc59b182204f3f8862506f568f754d, State: Running, Role: LEADER
I20260929 17:53:24.611194 5817 consensus_queue.cc:237] T 00000000000000000000000000000000 P cefc59b182204f3f8862506f568f754d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cefc59b182204f3f8862506f568f754d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40979 } }
I20260929 17:53:24.611752 5812 sys_catalog.cc:564] T 00000000000000000000000000000000 P cefc59b182204f3f8862506f568f754d [sys.catalog]: configured and running, proceeding with master startup.
I20260929 17:53:24.612726 5818 sys_catalog.cc:455] T 00000000000000000000000000000000 P cefc59b182204f3f8862506f568f754d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cefc59b182204f3f8862506f568f754d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cefc59b182204f3f8862506f568f754d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40979 } } }
I20260929 17:53:24.612854 5818 sys_catalog.cc:458] T 00000000000000000000000000000000 P cefc59b182204f3f8862506f568f754d [sys.catalog]: This master's current role is: LEADER
I20260929 17:53:24.613129 5819 sys_catalog.cc:455] T 00000000000000000000000000000000 P cefc59b182204f3f8862506f568f754d [sys.catalog]: SysCatalogTable state changed. Reason: New leader cefc59b182204f3f8862506f568f754d. Latest consensus state: current_term: 1 leader_uuid: "cefc59b182204f3f8862506f568f754d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cefc59b182204f3f8862506f568f754d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40979 } } }
I20260929 17:53:24.613219 5819 sys_catalog.cc:458] T 00000000000000000000000000000000 P cefc59b182204f3f8862506f568f754d [sys.catalog]: This master's current role is: LEADER
I20260929 17:53:24.613336 5826 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 17:53:24.616330 5826 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 17:53:24.621816 5826 catalog_manager.cc:1347] Generated new cluster ID: 84d45e3363244d79881d5b18fcf97c01
I20260929 17:53:24.621896 5826 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 17:53:24.632927 5826 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 17:53:24.634220 5826 catalog_manager.cc:1504] Loading token signing keys...
I20260929 17:53:24.648590 5826 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cefc59b182204f3f8862506f568f754d: Generated new TSK 0
I20260929 17:53:24.649384 5826 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 17:53:24.671048 5815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:24.665677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38628 (local address 127.0.98.62:40979)
0929 17:53:24.665802 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:24.665806 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:24.665976 (+ 170us) server_negotiation.cc:407] Connection header received
0929 17:53:24.666118 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:24.666121 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:24.666180 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:24.666291 (+ 111us) server_negotiation.cc:226] Negotiated authn=JWT
0929 17:53:24.666533 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:24.667150 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:24.668378 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:24.668606 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:24.668685 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:24.669034 (+ 349us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0929 17:53:24.670713 (+ 1679us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43295/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0929 17:53:24.670930 (+ 217us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:38628: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43295/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":52}
W20260929 17:53:24.671114 5815 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:38628: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43295/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260929 17:53:24.674024 5841 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:53:24.665551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40979 (local address 127.0.0.1:38628)
0929 17:53:24.665860 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:24.665875 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 17:53:24.665991 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:53:24.666315 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:53:24.666318 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:53:24.666326 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0929 17:53:24.666439 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:24.666445 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:24.667252 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:24.667256 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:24.668244 (+ 988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:24.668254 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:24.668692 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:24.668716 (+ 24us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:24.668807 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:24.668836 (+ 29us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0929 17:53:24.670873 (+ 2037us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43295/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0929 17:53:24.673874 (+ 3001us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:40979: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43295/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":216,"mutex_wait_us":2754,"thread_start_us":104,"threads_started":1}
I20260929 17:53:24.674755 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskup3Soe/build/debug/bin/kudu with pid 5745
E20260929 17:53:24.682996 5842 webserver.cc:533] Webserver: sslize failed:
E20260929 17:53:24.683070 5842 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-29T17:53:24Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (307 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-29T17:53:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T17:53:24Z Disabled control of system clock
I20260929 17:53:24.720445 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskup3Soe/build/debug/bin/kudu
/tmp/dist-test-taskup3Soe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790704375975685-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790704375975685-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790704375975685-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790704375975685-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38191
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:35191
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38191
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790704375975685-392-0/testchainca.cert with env {}
W20260929 17:53:24.827405 5846 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:53:24.827661 5846 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:53:24.827697 5846 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:53:24.831624 5846 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 17:53:24.831683 5846 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:53:24.831697 5846 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:53:24.831712 5846 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 17:53:24.831727 5846 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260929 17:53:24.831748 5846 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260929 17:53:24.831765 5846 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260929 17:53:24.837024 5846 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35191
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790704375975685-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790704375975685-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38191
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38191
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790704375975685-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790704375975685-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5846
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790704375975685-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:53:24.837989 5846 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:53:24.838971 5846 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:53:24.847002 5851 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:53:24.847002 5854 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:53:24.847092 5852 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:53:24.847208 5846 server_base.cc:1034] running on GCE node
I20260929 17:53:24.847867 5846 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:53:24.848950 5846 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:53:24.850126 5846 hybrid_clock.cc:648] HybridClock initialized: now 1790704404850107 us; error 30 us; skew 500 ppm
I20260929 17:53:24.851852 5846 webserver.cc:460] Webserver started at http://127.0.98.62:35001/ using document root <none> and password file <none>
I20260929 17:53:24.852368 5846 fs_manager.cc:359] Metadata directory not provided
I20260929 17:53:24.852424 5846 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:53:24.852584 5846 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:53:24.854600 5846 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790704375975685-392-0/minicluster-data/master-0/data/instance:
uuid: "72a36a5990674462ace8aaeb9889f138"
format_stamp: "Formatted at 2026-09-29 17:53:24 on dist-test-slave-53nh"
server_key: "9d406dc268a6fe80c83f5fea94db5ea5"
server_key_iv: "b73e338a6df0cb7acdaeed4c5e8fa285"
server_key_version: "encryptionkey@0"
I20260929 17:53:24.855089 5846 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790704375975685-392-0/minicluster-data/master-0/wal/instance:
uuid: "72a36a5990674462ace8aaeb9889f138"
format_stamp: "Formatted at 2026-09-29 17:53:24 on dist-test-slave-53nh"
server_key: "9d406dc268a6fe80c83f5fea94db5ea5"
server_key_iv: "b73e338a6df0cb7acdaeed4c5e8fa285"
server_key_version: "encryptionkey@0"
I20260929 17:53:24.858548 5846 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260929 17:53:24.860777 5860 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:53:24.862002 5846 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 17:53:24.862110 5846 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790704375975685-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790704375975685-392-0/minicluster-data/master-0/wal
uuid: "72a36a5990674462ace8aaeb9889f138"
format_stamp: "Formatted at 2026-09-29 17:53:24 on dist-test-slave-53nh"
server_key: "9d406dc268a6fe80c83f5fea94db5ea5"
server_key_iv: "b73e338a6df0cb7acdaeed4c5e8fa285"
server_key_version: "encryptionkey@0"
I20260929 17:53:24.862205 5846 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790704375975685-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790704375975685-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790704375975685-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:53:24.878778 5846 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:53:24.879434 5846 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:53:24.879596 5846 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:53:24.886124 5846 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38191
I20260929 17:53:24.886140 5912 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38191 every 8 connection(s)
I20260929 17:53:24.887009 5846 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790704375975685-392-0/minicluster-data/master-0/data/info.pb
I20260929 17:53:24.889701 5913 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:53:24.895119 5913 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 72a36a5990674462ace8aaeb9889f138: Bootstrap starting.
I20260929 17:53:24.895943 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskup3Soe/build/debug/bin/kudu as pid 5846
I20260929 17:53:24.896037 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790704375975685-392-0/minicluster-data/master-0/wal/instance
I20260929 17:53:24.896288 392 external_mini_cluster.cc:1428] Setting key b76a47e8428cd4aae21575c0bef1748f
I20260929 17:53:24.897795 5913 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 72a36a5990674462ace8aaeb9889f138: Neither blocks nor log segments found. Creating new log.
I20260929 17:53:24.898491 5913 log.cc:824] T 00000000000000000000000000000000 P 72a36a5990674462ace8aaeb9889f138: Log is configured to *not* fsync() on all Append() calls
I20260929 17:53:24.900213 5913 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 72a36a5990674462ace8aaeb9889f138: No bootstrap required, opened a new log
I20260929 17:53:24.903599 5913 raft_consensus.cc:348] T 00000000000000000000000000000000 P 72a36a5990674462ace8aaeb9889f138 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72a36a5990674462ace8aaeb9889f138" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38191 } }
I20260929 17:53:24.903822 5913 raft_consensus.cc:374] T 00000000000000000000000000000000 P 72a36a5990674462ace8aaeb9889f138 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:53:24.903874 5913 raft_consensus.cc:729] T 00000000000000000000000000000000 P 72a36a5990674462ace8aaeb9889f138 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72a36a5990674462ace8aaeb9889f138, State: Initialized, Role: FOLLOWER
I20260929 17:53:24.904280 5913 consensus_queue.cc:260] T 00000000000000000000000000000000 P 72a36a5990674462ace8aaeb9889f138 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72a36a5990674462ace8aaeb9889f138" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38191 } }
I20260929 17:53:24.904417 5913 raft_consensus.cc:388] T 00000000000000000000000000000000 P 72a36a5990674462ace8aaeb9889f138 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 17:53:24.904477 5913 raft_consensus.cc:482] T 00000000000000000000000000000000 P 72a36a5990674462ace8aaeb9889f138 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 17:53:24.904570 5913 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 72a36a5990674462ace8aaeb9889f138 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:53:24.905037 5916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:24.897264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44136 (local address 127.0.98.62:38191)
0929 17:53:24.897891 (+ 627us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:24.897908 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:24.897952 (+ 44us) server_negotiation.cc:407] Connection header received
0929 17:53:24.898555 (+ 603us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:24.898571 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:24.898980 (+ 409us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:24.899272 (+ 292us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:53:24.899694 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:24.900674 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:24.901549 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:24.902004 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:24.902143 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:24.903052 (+ 909us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:53:24.903059 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:53:24.903071 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 17:53:24.903448 (+ 377us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:53:24.903505 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 17:53:24.903525 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:53:24.903644 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0929 17:53:24.903787 (+ 143us) server_negotiation.cc:299] Negotiation successful
0929 17:53:24.904171 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":278,"thread_start_us":129,"threads_started":1}
I20260929 17:53:24.905547 5913 raft_consensus.cc:504] T 00000000000000000000000000000000 P 72a36a5990674462ace8aaeb9889f138 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72a36a5990674462ace8aaeb9889f138" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38191 } }
I20260929 17:53:24.905884 5913 leader_election.cc:302] T 00000000000000000000000000000000 P 72a36a5990674462ace8aaeb9889f138 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 72a36a5990674462ace8aaeb9889f138; no voters:
I20260929 17:53:24.906168 5913 leader_election.cc:288] T 00000000000000000000000000000000 P 72a36a5990674462ace8aaeb9889f138 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 17:53:24.906256 5918 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 72a36a5990674462ace8aaeb9889f138 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 17:53:24.906461 5918 raft_consensus.cc:686] T 00000000000000000000000000000000 P 72a36a5990674462ace8aaeb9889f138 [term 1 LEADER]: Becoming Leader. State: Replica: 72a36a5990674462ace8aaeb9889f138, State: Running, Role: LEADER
I20260929 17:53:24.906759 5918 consensus_queue.cc:237] T 00000000000000000000000000000000 P 72a36a5990674462ace8aaeb9889f138 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72a36a5990674462ace8aaeb9889f138" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38191 } }
I20260929 17:53:24.907398 5913 sys_catalog.cc:564] T 00000000000000000000000000000000 P 72a36a5990674462ace8aaeb9889f138 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 17:53:24.907815 5918 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72a36a5990674462ace8aaeb9889f138 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 72a36a5990674462ace8aaeb9889f138. Latest consensus state: current_term: 1 leader_uuid: "72a36a5990674462ace8aaeb9889f138" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72a36a5990674462ace8aaeb9889f138" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38191 } } }
I20260929 17:53:24.907918 5918 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72a36a5990674462ace8aaeb9889f138 [sys.catalog]: This master's current role is: LEADER
I20260929 17:53:24.907946 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 17:53:24.908473 5919 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72a36a5990674462ace8aaeb9889f138 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "72a36a5990674462ace8aaeb9889f138" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72a36a5990674462ace8aaeb9889f138" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38191 } } }
I20260929 17:53:24.908550 5919 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72a36a5990674462ace8aaeb9889f138 [sys.catalog]: This master's current role is: LEADER
I20260929 17:53:24.908790 5922 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 17:53:24.912177 5922 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 17:53:24.917068 5922 catalog_manager.cc:1347] Generated new cluster ID: 2d73e57a5d6441f7a1928b91edd99e42
I20260929 17:53:24.917125 5922 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 17:53:24.938977 5922 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 17:53:24.940114 5922 catalog_manager.cc:1504] Loading token signing keys...
I20260929 17:53:24.946115 5922 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 72a36a5990674462ace8aaeb9889f138: Generated new TSK 0
I20260929 17:53:24.946826 5922 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 17:53:25.041775 5916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:24.955970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44146 (local address 127.0.98.62:38191)
0929 17:53:24.956089 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:24.956094 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:24.956225 (+ 131us) server_negotiation.cc:407] Connection header received
0929 17:53:24.956372 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:24.956376 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:24.956429 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:24.956539 (+ 110us) server_negotiation.cc:226] Negotiated authn=JWT
0929 17:53:24.956692 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:24.957555 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:24.958355 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:24.958709 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:24.958814 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:24.959035 (+ 221us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0929 17:53:25.041323 (+ 82288us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0929 17:53:25.041624 (+ 301us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:44146: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":42}
W20260929 17:53:25.041863 5916 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:44146: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260929 17:53:25.041898 5942 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:53:24.955871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38191 (local address 127.0.0.1:44146)
0929 17:53:24.956151 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:24.956162 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 17:53:24.956260 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:53:24.956530 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:53:24.956532 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:53:24.956538 (+ 6us) client_negotiation.cc:189] Negotiated authn=JWT
0929 17:53:24.956625 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:24.956628 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:24.957648 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:24.957651 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:24.958260 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:24.958266 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:24.958794 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:24.958811 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:24.958868 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:24.958890 (+ 22us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0929 17:53:25.041555 (+ 82665us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0929 17:53:25.041741 (+ 186us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:38191: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":103,"threads_started":1}
I20260929 17:53:25.042688 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskup3Soe/build/debug/bin/kudu with pid 5846
2026-09-29T17:53:25Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (347 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (119 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (119 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704375975685-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 17:53:25 dist-test-slave-53nh krb5kdc[5954](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 17:53:25 dist-test-slave-53nh krb5kdc[5954](info): set up 2 sockets
Sep 29 17:53:25 dist-test-slave-53nh krb5kdc[5954](info): commencing operation
krb5kdc: starting...
W20260929 17:53:26.233362 392 mini_kdc.cc:121] Time spent starting KDC: real 0.933s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704375975685-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704375975685-392-0/krb5kdc/test-user.keytab.
Sep 29 17:53:26 dist-test-slave-53nh krb5kdc[5954](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704406, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T17:53:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T17:53:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704375975685-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704375975685-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 17:53:26.349805 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskup3Soe/build/debug/bin/kudu
/tmp/dist-test-taskup3Soe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704375975685-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704375975685-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704375975685-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704375975685-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38831
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704375975685-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38831
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704375975685-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704375975685-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704375975685-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:53:26.458596 5967 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:53:26.458901 5967 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:53:26.458967 5967 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:53:26.463096 5967 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 17:53:26.463177 5967 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:53:26.463209 5967 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:53:26.463236 5967 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 17:53:26.463268 5967 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 17:53:26.468787 5967 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704375975685-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704375975685-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38831
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704375975685-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38831
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704375975685-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5967
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704375975685-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:53:26.469940 5967 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:53:26.471004 5967 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:53:26.481154 5972 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:53:26.481578 5973 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:53:26.481635 5967 server_base.cc:1034] running on GCE node
W20260929 17:53:26.481719 5975 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:53:26.482375 5967 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:53:26.483539 5967 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:53:26.484736 5967 hybrid_clock.cc:648] HybridClock initialized: now 1790704406484705 us; error 57 us; skew 500 ppm
Sep 29 17:53:26 dist-test-slave-53nh krb5kdc[5954](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704406, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:53:26.487596 5967 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 17:53:26.488562 5967 webserver.cc:460] Webserver started at http://127.0.98.62:32793/ using document root <none> and password file <none>
I20260929 17:53:26.489176 5967 fs_manager.cc:359] Metadata directory not provided
I20260929 17:53:26.489257 5967 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:53:26.489465 5967 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:53:26.491559 5967 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704375975685-392-0/minicluster-data/master-0/data/instance:
uuid: "c5f8e4e28a2c4a4e90bdba7df69502b1"
format_stamp: "Formatted at 2026-09-29 17:53:26 on dist-test-slave-53nh"
server_key: "093abc3398f64e4356cb328f6c7ac93f"
server_key_iv: "e2de11b98239017573235ac9274f8ed3"
server_key_version: "encryptionkey@0"
I20260929 17:53:26.492144 5967 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704375975685-392-0/minicluster-data/master-0/wal/instance:
uuid: "c5f8e4e28a2c4a4e90bdba7df69502b1"
format_stamp: "Formatted at 2026-09-29 17:53:26 on dist-test-slave-53nh"
server_key: "093abc3398f64e4356cb328f6c7ac93f"
server_key_iv: "e2de11b98239017573235ac9274f8ed3"
server_key_version: "encryptionkey@0"
I20260929 17:53:26.496714 5967 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260929 17:53:26.499402 5982 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:53:26.500702 5967 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 17:53:26.500815 5967 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704375975685-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704375975685-392-0/minicluster-data/master-0/wal
uuid: "c5f8e4e28a2c4a4e90bdba7df69502b1"
format_stamp: "Formatted at 2026-09-29 17:53:26 on dist-test-slave-53nh"
server_key: "093abc3398f64e4356cb328f6c7ac93f"
server_key_iv: "e2de11b98239017573235ac9274f8ed3"
server_key_version: "encryptionkey@0"
I20260929 17:53:26.500912 5967 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704375975685-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704375975685-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704375975685-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:53:26.510982 5967 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:53:26.513851 5967 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:53:26.514014 5967 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:53:26.520190 5967 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38831
I20260929 17:53:26.520427 6034 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38831 every 8 connection(s)
I20260929 17:53:26.521127 5967 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704375975685-392-0/minicluster-data/master-0/data/info.pb
I20260929 17:53:26.523682 6035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:53:26.525575 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskup3Soe/build/debug/bin/kudu as pid 5967
I20260929 17:53:26.525682 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704375975685-392-0/minicluster-data/master-0/wal/instance
I20260929 17:53:26.525925 392 external_mini_cluster.cc:1428] Setting key 23109619b2dc64697ce118a54650e315
I20260929 17:53:26.529902 6035 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c5f8e4e28a2c4a4e90bdba7df69502b1: Bootstrap starting.
I20260929 17:53:26.532263 6035 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c5f8e4e28a2c4a4e90bdba7df69502b1: Neither blocks nor log segments found. Creating new log.
I20260929 17:53:26.533090 6035 log.cc:824] T 00000000000000000000000000000000 P c5f8e4e28a2c4a4e90bdba7df69502b1: Log is configured to *not* fsync() on all Append() calls
Sep 29 17:53:26 dist-test-slave-53nh krb5kdc[5954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704406, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 17:53:26.535058 6035 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c5f8e4e28a2c4a4e90bdba7df69502b1: No bootstrap required, opened a new log
I20260929 17:53:26.538663 6035 raft_consensus.cc:348] T 00000000000000000000000000000000 P c5f8e4e28a2c4a4e90bdba7df69502b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5f8e4e28a2c4a4e90bdba7df69502b1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38831 } }
I20260929 17:53:26.538880 6035 raft_consensus.cc:374] T 00000000000000000000000000000000 P c5f8e4e28a2c4a4e90bdba7df69502b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:53:26.538934 6035 raft_consensus.cc:729] T 00000000000000000000000000000000 P c5f8e4e28a2c4a4e90bdba7df69502b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5f8e4e28a2c4a4e90bdba7df69502b1, State: Initialized, Role: FOLLOWER
I20260929 17:53:26.539338 6038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:26.527115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40578 (local address 127.0.98.62:38831)
0929 17:53:26.527469 (+ 354us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:26.527483 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:26.527516 (+ 33us) server_negotiation.cc:407] Connection header received
0929 17:53:26.528143 (+ 627us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:26.528159 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:26.528474 (+ 315us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:26.528743 (+ 269us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:53:26.529615 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:26.530594 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:26.531722 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:26.532216 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:26.532370 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:26.534954 (+ 2584us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:53:26.534959 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:53:26.534972 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:53:26.534996 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:53:26.536564 (+ 1568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:53:26.537102 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:53:26.537108 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:53:26.537113 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:53:26.537227 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:53:26.537494 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:53:26.537496 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:53:26.537497 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:53:26.537865 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:53:26.538016 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0929 17:53:26.538281 (+ 265us) server_negotiation.cc:299] Negotiation successful
0929 17:53:26.538553 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":231,"thread_start_us":94,"threads_started":1}
I20260929 17:53:26.539341 6035 consensus_queue.cc:260] T 00000000000000000000000000000000 P c5f8e4e28a2c4a4e90bdba7df69502b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5f8e4e28a2c4a4e90bdba7df69502b1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38831 } }
I20260929 17:53:26.539537 6035 raft_consensus.cc:388] T 00000000000000000000000000000000 P c5f8e4e28a2c4a4e90bdba7df69502b1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 17:53:26.539592 6035 raft_consensus.cc:482] T 00000000000000000000000000000000 P c5f8e4e28a2c4a4e90bdba7df69502b1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 17:53:26.539683 6035 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c5f8e4e28a2c4a4e90bdba7df69502b1 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:53:26.540504 6035 raft_consensus.cc:504] T 00000000000000000000000000000000 P c5f8e4e28a2c4a4e90bdba7df69502b1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5f8e4e28a2c4a4e90bdba7df69502b1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38831 } }
I20260929 17:53:26.540827 6035 leader_election.cc:302] T 00000000000000000000000000000000 P c5f8e4e28a2c4a4e90bdba7df69502b1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c5f8e4e28a2c4a4e90bdba7df69502b1; no voters:
I20260929 17:53:26.541131 6035 leader_election.cc:288] T 00000000000000000000000000000000 P c5f8e4e28a2c4a4e90bdba7df69502b1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 17:53:26.541214 6040 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c5f8e4e28a2c4a4e90bdba7df69502b1 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 17:53:26.541375 6040 raft_consensus.cc:686] T 00000000000000000000000000000000 P c5f8e4e28a2c4a4e90bdba7df69502b1 [term 1 LEADER]: Becoming Leader. State: Replica: c5f8e4e28a2c4a4e90bdba7df69502b1, State: Running, Role: LEADER
I20260929 17:53:26.541654 6040 consensus_queue.cc:237] T 00000000000000000000000000000000 P c5f8e4e28a2c4a4e90bdba7df69502b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5f8e4e28a2c4a4e90bdba7df69502b1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38831 } }
I20260929 17:53:26.542359 6035 sys_catalog.cc:564] T 00000000000000000000000000000000 P c5f8e4e28a2c4a4e90bdba7df69502b1 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 17:53:26.543320 6042 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5f8e4e28a2c4a4e90bdba7df69502b1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c5f8e4e28a2c4a4e90bdba7df69502b1. Latest consensus state: current_term: 1 leader_uuid: "c5f8e4e28a2c4a4e90bdba7df69502b1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5f8e4e28a2c4a4e90bdba7df69502b1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38831 } } }
I20260929 17:53:26.543452 6042 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5f8e4e28a2c4a4e90bdba7df69502b1 [sys.catalog]: This master's current role is: LEADER
I20260929 17:53:26.543745 6041 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5f8e4e28a2c4a4e90bdba7df69502b1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c5f8e4e28a2c4a4e90bdba7df69502b1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5f8e4e28a2c4a4e90bdba7df69502b1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38831 } } }
I20260929 17:53:26.543861 6041 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5f8e4e28a2c4a4e90bdba7df69502b1 [sys.catalog]: This master's current role is: LEADER
I20260929 17:53:26.543960 6049 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 17:53:26.546965 6049 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 17:53:26.552338 6049 catalog_manager.cc:1347] Generated new cluster ID: 858000b1466c4783a9e8a1dcb4643603
I20260929 17:53:26.552408 6049 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 17:53:26.570039 6049 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 17:53:26.571044 6049 catalog_manager.cc:1504] Loading token signing keys...
I20260929 17:53:26.576514 6049 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c5f8e4e28a2c4a4e90bdba7df69502b1: Generated new TSK 0
I20260929 17:53:26.577237 6049 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704375975685-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704375975685-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260929 17:53:26.606174 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskup3Soe/build/debug/bin/kudu
/tmp/dist-test-taskup3Soe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704375975685-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704375975685-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704375975685-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704375975685-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38831
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704375975685-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704375975685-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704375975685-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704375975685-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:53:26.741245 6061 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:53:26.741514 6061 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:53:26.741557 6061 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:53:26.745702 6061 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:53:26.745766 6061 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:53:26.745826 6061 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 17:53:26.750967 6061 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704375975685-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704375975685-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704375975685-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704375975685-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38831
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6061
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704375975685-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:53:26.751981 6061 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:53:26.752939 6061 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 17:53:26.762260 6061 server_base.cc:1034] running on GCE node
W20260929 17:53:26.762223 6067 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:53:26.762217 6069 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:53:26.762224 6066 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:53:26.762815 6061 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:53:26.763429 6061 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:53:26.764583 6061 hybrid_clock.cc:648] HybridClock initialized: now 1790704406764562 us; error 38 us; skew 500 ppm
Sep 29 17:53:26 dist-test-slave-53nh krb5kdc[5954](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704406, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:53:26.767279 6061 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260929 17:53:26.768077 6061 webserver.cc:460] Webserver started at http://127.0.98.1:41399/ using document root <none> and password file <none>
I20260929 17:53:26.768592 6061 fs_manager.cc:359] Metadata directory not provided
I20260929 17:53:26.768649 6061 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:53:26.768826 6061 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:53:26.770891 6061 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704375975685-392-0/minicluster-data/ts-0/data/instance:
uuid: "2b121defd21447558b65b830090b7bc5"
format_stamp: "Formatted at 2026-09-29 17:53:26 on dist-test-slave-53nh"
server_key: "23509412b57d4b818f6e10c69231715e"
server_key_iv: "b015306b0829cdd39f94c14cb4318039"
server_key_version: "encryptionkey@0"
I20260929 17:53:26.771382 6061 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704375975685-392-0/minicluster-data/ts-0/wal/instance:
uuid: "2b121defd21447558b65b830090b7bc5"
format_stamp: "Formatted at 2026-09-29 17:53:26 on dist-test-slave-53nh"
server_key: "23509412b57d4b818f6e10c69231715e"
server_key_iv: "b015306b0829cdd39f94c14cb4318039"
server_key_version: "encryptionkey@0"
I20260929 17:53:26.774870 6061 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 17:53:26.777161 6076 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:53:26.778435 6061 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 17:53:26.778546 6061 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704375975685-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704375975685-392-0/minicluster-data/ts-0/wal
uuid: "2b121defd21447558b65b830090b7bc5"
format_stamp: "Formatted at 2026-09-29 17:53:26 on dist-test-slave-53nh"
server_key: "23509412b57d4b818f6e10c69231715e"
server_key_iv: "b015306b0829cdd39f94c14cb4318039"
server_key_version: "encryptionkey@0"
I20260929 17:53:26.778656 6061 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704375975685-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704375975685-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704375975685-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:53:26.792919 6061 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:53:26.796114 6061 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:53:26.796293 6061 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:53:26.796855 6061 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 17:53:26.797706 6061 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:53:26.797755 6061 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:53:26.797806 6061 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:53:26.797832 6061 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 17:53:26.805884 6061 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38349
I20260929 17:53:26.805922 6189 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38349 every 8 connection(s)
I20260929 17:53:26.806756 6061 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704375975685-392-0/minicluster-data/ts-0/data/info.pb
I20260929 17:53:26.812919 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskup3Soe/build/debug/bin/kudu as pid 6061
I20260929 17:53:26.813066 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704375975685-392-0/minicluster-data/ts-0/wal/instance
I20260929 17:53:26.813324 392 external_mini_cluster.cc:1428] Setting key 097abe389f5761aba5443aecb81b5b74
Sep 29 17:53:26 dist-test-slave-53nh krb5kdc[5954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704406, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 17:53:26.819725 6038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:26.808448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45441 (local address 127.0.98.62:38831)
0929 17:53:26.808581 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:26.808585 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:26.809473 (+ 888us) server_negotiation.cc:407] Connection header received
0929 17:53:26.810804 (+ 1331us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:26.810808 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:26.810861 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:26.810932 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:53:26.812049 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:26.812588 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:26.813808 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:26.814038 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:26.814123 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:26.816350 (+ 2227us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:53:26.816353 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:53:26.816355 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:53:26.816377 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:53:26.817627 (+ 1250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:53:26.818240 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:53:26.818243 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:53:26.818244 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:53:26.818313 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:53:26.818666 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:53:26.818668 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:53:26.818669 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:53:26.818826 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:53:26.818892 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0929 17:53:26.819506 (+ 614us) server_negotiation.cc:299] Negotiation successful
0929 17:53:26.819606 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 17:53:26.820870 6192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:26.808846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38831 (local address 127.0.98.1:45441)
0929 17:53:26.809334 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:26.809376 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 17:53:26.810602 (+ 1226us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:53:26.811104 (+ 502us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:53:26.811111 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:53:26.811455 (+ 344us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:53:26.811912 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:26.811926 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:26.812717 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:26.812720 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:26.813698 (+ 978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:26.813710 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:26.814169 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:26.814173 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:26.814332 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:26.814862 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:53:26.814878 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:53:26.816202 (+ 1324us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:53:26.817750 (+ 1548us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:53:26.817757 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:53:26.817772 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:53:26.818129 (+ 357us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:53:26.818398 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:53:26.818401 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:53:26.818403 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:53:26.818585 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:53:26.818917 (+ 332us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:53:26.818924 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:53:26.819242 (+ 318us) client_negotiation.cc:769] Sending connection context
0929 17:53:26.819459 (+ 217us) client_negotiation.cc:240] Negotiation successful
0929 17:53:26.819818 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":147,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260929 17:53:26.822074 6190 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38831
I20260929 17:53:26.822279 6190 heartbeater.cc:464] Registering TS with master...
I20260929 17:53:26.822706 6190 heartbeater.cc:511] Master 127.0.98.62:38831 requested a full tablet report, sending...
I20260929 17:53:26.824163 5999 ts_manager.cc:194] Registered new tserver with Master: 2b121defd21447558b65b830090b7bc5 (127.0.98.1:38349)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260929 17:53:26.825598 5999 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:45441
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704375975685-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704375975685-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260929 17:53:26.840240 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskup3Soe/build/debug/bin/kudu
/tmp/dist-test-taskup3Soe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704375975685-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704375975685-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704375975685-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704375975685-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38831
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704375975685-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704375975685-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704375975685-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704375975685-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:53:26.948982 6195 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:53:26.949266 6195 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:53:26.949309 6195 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:53:26.953320 6195 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:53:26.953382 6195 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:53:26.953452 6195 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 17:53:26.958806 6195 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704375975685-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704375975685-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704375975685-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704375975685-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38831
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6195
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704375975685-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:53:26.959827 6195 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:53:26.960830 6195 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:53:26.970247 6203 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:53:26.970216 6201 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:53:26.970216 6200 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:53:26.970438 6195 server_base.cc:1034] running on GCE node
I20260929 17:53:26.970950 6195 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:53:26.971624 6195 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:53:26.972796 6195 hybrid_clock.cc:648] HybridClock initialized: now 1790704406972776 us; error 41 us; skew 500 ppm
Sep 29 17:53:26 dist-test-slave-53nh krb5kdc[5954](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704406, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:53:26.979281 6195 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260929 17:53:26.980217 6195 webserver.cc:460] Webserver started at http://127.0.98.2:34515/ using document root <none> and password file <none>
I20260929 17:53:26.980741 6195 fs_manager.cc:359] Metadata directory not provided
I20260929 17:53:26.980796 6195 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:53:26.980973 6195 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:53:26.983162 6195 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704375975685-392-0/minicluster-data/ts-1/data/instance:
uuid: "f346cbb6cb124bcf85f15813b838e82c"
format_stamp: "Formatted at 2026-09-29 17:53:26 on dist-test-slave-53nh"
server_key: "f1265cd18e8ef47c1eab2cb7613878ab"
server_key_iv: "b5087de5ddfaa3f68067e2e54b124b24"
server_key_version: "encryptionkey@0"
I20260929 17:53:26.983692 6195 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704375975685-392-0/minicluster-data/ts-1/wal/instance:
uuid: "f346cbb6cb124bcf85f15813b838e82c"
format_stamp: "Formatted at 2026-09-29 17:53:26 on dist-test-slave-53nh"
server_key: "f1265cd18e8ef47c1eab2cb7613878ab"
server_key_iv: "b5087de5ddfaa3f68067e2e54b124b24"
server_key_version: "encryptionkey@0"
I20260929 17:53:26.987675 6195 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.005s
I20260929 17:53:26.990150 6210 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:53:26.991400 6195 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 17:53:26.991505 6195 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704375975685-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704375975685-392-0/minicluster-data/ts-1/wal
uuid: "f346cbb6cb124bcf85f15813b838e82c"
format_stamp: "Formatted at 2026-09-29 17:53:26 on dist-test-slave-53nh"
server_key: "f1265cd18e8ef47c1eab2cb7613878ab"
server_key_iv: "b5087de5ddfaa3f68067e2e54b124b24"
server_key_version: "encryptionkey@0"
I20260929 17:53:26.991616 6195 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704375975685-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704375975685-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704375975685-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:53:27.006927 6195 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:53:27.010432 6195 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:53:27.010613 6195 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:53:27.011222 6195 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 17:53:27.012077 6195 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:53:27.012126 6195 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:53:27.012166 6195 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:53:27.012180 6195 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 17:53:27.021173 6195 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45709
I20260929 17:53:27.022346 6195 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704375975685-392-0/minicluster-data/ts-1/data/info.pb
I20260929 17:53:27.022543 6323 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45709 every 8 connection(s)
I20260929 17:53:27.026352 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskup3Soe/build/debug/bin/kudu as pid 6195
I20260929 17:53:27.026489 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704375975685-392-0/minicluster-data/ts-1/wal/instance
I20260929 17:53:27.026804 392 external_mini_cluster.cc:1428] Setting key db0c76fba4a4de563481069d4b125281
Sep 29 17:53:27 dist-test-slave-53nh krb5kdc[5954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704406, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260929 17:53:27.038019 6038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:27.024569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35557 (local address 127.0.98.62:38831)
0929 17:53:27.024751 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:27.024756 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:27.025817 (+ 1061us) server_negotiation.cc:407] Connection header received
0929 17:53:27.027527 (+ 1710us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:27.027530 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:27.027584 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:27.027666 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:53:27.029384 (+ 1718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:27.029926 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:27.031137 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:27.031369 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:27.031463 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:27.034328 (+ 2865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:53:27.034333 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:53:27.034336 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:53:27.034370 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:53:27.035957 (+ 1587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:53:27.036503 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:53:27.036506 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:53:27.036508 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:53:27.036585 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:53:27.036873 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:53:27.036875 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:53:27.036876 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:53:27.037104 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:53:27.037190 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0929 17:53:27.037772 (+ 582us) server_negotiation.cc:299] Negotiation successful
0929 17:53:27.037894 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260929 17:53:27.039280 6326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:27.025062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38831 (local address 127.0.98.2:35557)
0929 17:53:27.025642 (+ 580us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:27.025706 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0929 17:53:27.027279 (+ 1573us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:53:27.027821 (+ 542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:53:27.027830 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:53:27.028355 (+ 525us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:53:27.029204 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:27.029230 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:27.030052 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:27.030056 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:27.031009 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:27.031019 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:27.031516 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:27.031519 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:27.031721 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:27.032372 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:53:27.032390 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:53:27.034131 (+ 1741us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:53:27.036080 (+ 1949us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:53:27.036087 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:53:27.036106 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:53:27.036394 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:53:27.036652 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:53:27.036654 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:53:27.036655 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:53:27.036803 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:53:27.037219 (+ 416us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:53:27.037227 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:53:27.037516 (+ 289us) client_negotiation.cc:769] Sending connection context
0929 17:53:27.037716 (+ 200us) client_negotiation.cc:240] Negotiation successful
0929 17:53:27.038011 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":133,"threads_started":1}
I20260929 17:53:27.040490 6324 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38831
I20260929 17:53:27.040732 6324 heartbeater.cc:464] Registering TS with master...
I20260929 17:53:27.041298 6324 heartbeater.cc:511] Master 127.0.98.62:38831 requested a full tablet report, sending...
I20260929 17:53:27.042439 5999 ts_manager.cc:194] Registered new tserver with Master: f346cbb6cb124bcf85f15813b838e82c (127.0.98.2:45709)
I20260929 17:53:27.043136 5999 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:35557
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704375975685-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704375975685-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260929 17:53:27.056727 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskup3Soe/build/debug/bin/kudu
/tmp/dist-test-taskup3Soe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704375975685-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704375975685-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704375975685-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704375975685-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38831
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704375975685-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704375975685-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704375975685-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704375975685-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:53:27.164460 6329 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:53:27.164738 6329 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:53:27.164798 6329 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:53:27.168797 6329 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:53:27.168857 6329 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:53:27.168937 6329 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 17:53:27.174129 6329 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704375975685-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704375975685-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704375975685-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704375975685-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38831
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6329
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704375975685-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:53:27.175226 6329 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:53:27.176388 6329 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:53:27.185427 6335 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:53:27.185479 6337 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:53:27.185438 6334 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:53:27.185854 6329 server_base.cc:1034] running on GCE node
I20260929 17:53:27.186256 6329 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:53:27.186852 6329 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:53:27.188016 6329 hybrid_clock.cc:648] HybridClock initialized: now 1790704407187988 us; error 40 us; skew 500 ppm
Sep 29 17:53:27 dist-test-slave-53nh krb5kdc[5954](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704407, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:53:27.190613 6329 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260929 17:53:27.191428 6329 webserver.cc:460] Webserver started at http://127.0.98.3:34785/ using document root <none> and password file <none>
I20260929 17:53:27.191922 6329 fs_manager.cc:359] Metadata directory not provided
I20260929 17:53:27.191974 6329 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:53:27.192135 6329 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:53:27.194092 6329 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704375975685-392-0/minicluster-data/ts-2/data/instance:
uuid: "cf3f5ded14f349f7924d9700dfaeb75a"
format_stamp: "Formatted at 2026-09-29 17:53:27 on dist-test-slave-53nh"
server_key: "8f56c2e29b8b0152214ab046f14c4344"
server_key_iv: "f1d595bd3fca8eeaedfec77702256445"
server_key_version: "encryptionkey@0"
I20260929 17:53:27.194622 6329 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704375975685-392-0/minicluster-data/ts-2/wal/instance:
uuid: "cf3f5ded14f349f7924d9700dfaeb75a"
format_stamp: "Formatted at 2026-09-29 17:53:27 on dist-test-slave-53nh"
server_key: "8f56c2e29b8b0152214ab046f14c4344"
server_key_iv: "f1d595bd3fca8eeaedfec77702256445"
server_key_version: "encryptionkey@0"
I20260929 17:53:27.198282 6329 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 17:53:27.200680 6344 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:53:27.201912 6329 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 17:53:27.202023 6329 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704375975685-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704375975685-392-0/minicluster-data/ts-2/wal
uuid: "cf3f5ded14f349f7924d9700dfaeb75a"
format_stamp: "Formatted at 2026-09-29 17:53:27 on dist-test-slave-53nh"
server_key: "8f56c2e29b8b0152214ab046f14c4344"
server_key_iv: "f1d595bd3fca8eeaedfec77702256445"
server_key_version: "encryptionkey@0"
I20260929 17:53:27.202116 6329 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704375975685-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704375975685-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704375975685-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:53:27.222077 6329 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:53:27.225178 6329 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:53:27.225363 6329 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:53:27.225948 6329 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 17:53:27.226758 6329 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:53:27.226804 6329 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:53:27.226854 6329 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:53:27.226882 6329 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 17:53:27.235410 6329 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43831
I20260929 17:53:27.235445 6457 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43831 every 8 connection(s)
I20260929 17:53:27.236356 6329 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704375975685-392-0/minicluster-data/ts-2/data/info.pb
I20260929 17:53:27.242841 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskup3Soe/build/debug/bin/kudu as pid 6329
I20260929 17:53:27.242949 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704375975685-392-0/minicluster-data/ts-2/wal/instance
I20260929 17:53:27.243227 392 external_mini_cluster.cc:1428] Setting key a57ce8c8b1a12b780b609a6cdb66696e
Sep 29 17:53:27 dist-test-slave-53nh krb5kdc[5954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704407, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 17:53:27.250782 6038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:27.238369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40389 (local address 127.0.98.62:38831)
0929 17:53:27.238523 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:27.238526 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:27.239364 (+ 838us) server_negotiation.cc:407] Connection header received
0929 17:53:27.240826 (+ 1462us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:27.240830 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:27.240883 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:27.240970 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:53:27.242378 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:27.243047 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:27.244485 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:27.244778 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:27.244880 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:27.247395 (+ 2515us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:53:27.247398 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:53:27.247401 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:53:27.247427 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:53:27.248700 (+ 1273us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:53:27.249283 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:53:27.249286 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:53:27.249287 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:53:27.249366 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:53:27.249699 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:53:27.249702 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:53:27.249704 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:53:27.249908 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:53:27.249994 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0929 17:53:27.250537 (+ 543us) server_negotiation.cc:299] Negotiation successful
0929 17:53:27.250656 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 17:53:27.251572 6460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:27.238733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38831 (local address 127.0.98.3:40389)
0929 17:53:27.239195 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:27.239241 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 17:53:27.240547 (+ 1306us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:53:27.241240 (+ 693us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:53:27.241249 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:53:27.241674 (+ 425us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:53:27.242213 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:27.242228 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:27.243183 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:27.243187 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:27.244349 (+ 1162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:27.244363 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:27.244937 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:27.244940 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:27.245140 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:27.245762 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:53:27.245779 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:53:27.247225 (+ 1446us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:53:27.248818 (+ 1593us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:53:27.248823 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:53:27.248837 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:53:27.249162 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:53:27.249439 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:53:27.249441 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:53:27.249442 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:53:27.249580 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:53:27.250007 (+ 427us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:53:27.250013 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:53:27.250275 (+ 262us) client_negotiation.cc:769] Sending connection context
0929 17:53:27.250441 (+ 166us) client_negotiation.cc:240] Negotiation successful
0929 17:53:27.250712 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":116,"threads_started":1}
I20260929 17:53:27.252806 6458 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38831
I20260929 17:53:27.253078 6458 heartbeater.cc:464] Registering TS with master...
I20260929 17:53:27.253609 6458 heartbeater.cc:511] Master 127.0.98.62:38831 requested a full tablet report, sending...
I20260929 17:53:27.254746 5999 ts_manager.cc:194] Registered new tserver with Master: cf3f5ded14f349f7924d9700dfaeb75a (127.0.98.3:43831)
I20260929 17:53:27.255358 5999 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:40389
I20260929 17:53:27.257285 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704375975685-392-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704375975685-392-0/krb5kdc/test-admin.keytab.
Sep 29 17:53:27 dist-test-slave-53nh krb5kdc[5954](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704407, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:53:27.273370 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260929 17:53:27.275987 6465 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:53:27.274645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43696 (local address 127.0.98.1:38349)
0929 17:53:27.274954 (+ 309us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:27.274965 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:27.274988 (+ 23us) server_negotiation.cc:407] Connection header received
0929 17:53:27.275038 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:27.275043 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:27.275256 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:27.275373 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:53:27.275814 (+ 441us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43696: BlockingRecv error: recv got EOF from 127.0.0.1:43696 (error 108)
Metrics: {"server-negotiator.queue_time_us":208,"thread_start_us":116,"threads_started":1}
W20260929 17:53:27.276070 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704375975685-392-0/krb5kdc/krb5cc' of type FILE
Sep 29 17:53:27 dist-test-slave-53nh krb5kdc[5954](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704407, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:53:27.277109 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 29 17:53:27 dist-test-slave-53nh krb5kdc[5954](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704407, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:53:27.277998 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 29 17:53:27 dist-test-slave-53nh krb5kdc[5954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704407, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260929 17:53:27.287082 6465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:27.278834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43700 (local address 127.0.98.1:38349)
0929 17:53:27.278949 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:27.278954 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:27.279023 (+ 69us) server_negotiation.cc:407] Connection header received
0929 17:53:27.279150 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:27.279153 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:27.279195 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:27.279256 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:53:27.279764 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:27.280526 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:27.281425 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:27.281809 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:27.281901 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:27.284019 (+ 2118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:53:27.284028 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:53:27.284041 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:53:27.284064 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:53:27.285448 (+ 1384us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:53:27.285839 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:53:27.285845 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:53:27.285846 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:53:27.285921 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:53:27.286157 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:53:27.286160 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:53:27.286162 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:53:27.286403 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:53:27.286592 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0929 17:53:27.286789 (+ 197us) server_negotiation.cc:299] Negotiation successful
0929 17:53:27.286936 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
W20260929 17:53:27.290573 6465 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:53:27.289694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43702 (local address 127.0.98.1:38349)
0929 17:53:27.289802 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:27.289806 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:27.289871 (+ 65us) server_negotiation.cc:407] Connection header received
0929 17:53:27.289988 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:27.289991 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:27.290028 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:27.290088 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:53:27.290480 (+ 392us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43702: BlockingRecv error: recv got EOF from 127.0.0.1:43702 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260929 17:53:27.290880 392 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704375975685-392-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704375975685-392-0/krb5kdc/krb5cc)
Sep 29 17:53:27 dist-test-slave-53nh krb5kdc[5954](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704407, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:53:27.291766 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 29 17:53:27 dist-test-slave-53nh krb5kdc[5954](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704407, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:53:27.292541 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 29 17:53:27 dist-test-slave-53nh krb5kdc[5954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704407, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260929 17:53:27.299961 6465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:27.293354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43714 (local address 127.0.98.1:38349)
0929 17:53:27.293458 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:27.293461 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:27.293487 (+ 26us) server_negotiation.cc:407] Connection header received
0929 17:53:27.293583 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:27.293586 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:27.293617 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:27.293676 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:53:27.294093 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:27.294632 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:27.295415 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:27.295634 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:27.295703 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:27.297724 (+ 2021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:53:27.297727 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:53:27.297730 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:53:27.297750 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:53:27.298782 (+ 1032us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:53:27.299109 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:53:27.299111 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:53:27.299112 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:53:27.299173 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:53:27.299365 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:53:27.299367 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:53:27.299368 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:53:27.299514 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:53:27.299605 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0929 17:53:27.299765 (+ 160us) server_negotiation.cc:299] Negotiation successful
0929 17:53:27.299853 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
W20260929 17:53:27.302419 6465 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:53:27.301546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43726 (local address 127.0.98.1:38349)
0929 17:53:27.301672 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:27.301676 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:27.301746 (+ 70us) server_negotiation.cc:407] Connection header received
0929 17:53:27.301866 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:27.301869 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:27.301905 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:27.301965 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:53:27.302327 (+ 362us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43726: BlockingRecv error: recv got EOF from 127.0.0.1:43726 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
W20260929 17:53:27.302583 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790704375975685-392-0/krb5kdc/krb5cc' of type FILE
Sep 29 17:53:27 dist-test-slave-53nh krb5kdc[5954](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704407, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:53:27.303455 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 29 17:53:27 dist-test-slave-53nh krb5kdc[5954](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704407, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:53:27.304289 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 29 17:53:27 dist-test-slave-53nh krb5kdc[5954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704407, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260929 17:53:27.311887 6465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:27.305136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43736 (local address 127.0.98.1:38349)
0929 17:53:27.305262 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:27.305265 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:27.305277 (+ 12us) server_negotiation.cc:407] Connection header received
0929 17:53:27.305333 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:27.305337 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:27.305372 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:27.305433 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:53:27.305908 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:27.306392 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:27.307263 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:27.307459 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:27.307531 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:27.309504 (+ 1973us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:53:27.309507 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:53:27.309509 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:53:27.309528 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:53:27.310691 (+ 1163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:53:27.311021 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:53:27.311023 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:53:27.311024 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:53:27.311083 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:53:27.311269 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:53:27.311271 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:53:27.311272 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:53:27.311407 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:53:27.311491 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0929 17:53:27.311662 (+ 171us) server_negotiation.cc:299] Negotiation successful
0929 17:53:27.311744 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260929 17:53:27.312778 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskup3Soe/build/debug/bin/kudu with pid 6061
I20260929 17:53:27.318218 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskup3Soe/build/debug/bin/kudu with pid 6195
I20260929 17:53:27.324291 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskup3Soe/build/debug/bin/kudu with pid 6329
I20260929 17:53:27.328655 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskup3Soe/build/debug/bin/kudu with pid 5967
2026-09-29T17:53:27Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2048 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704375975685-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 17:53:27 dist-test-slave-53nh krb5kdc[6478](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 17:53:27 dist-test-slave-53nh krb5kdc[6478](info): set up 2 sockets
Sep 29 17:53:27 dist-test-slave-53nh krb5kdc[6478](info): commencing operation
krb5kdc: starting...
W20260929 17:53:28.273663 392 mini_kdc.cc:121] Time spent starting KDC: real 0.925s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704375975685-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704375975685-392-0/krb5kdc/test-user.keytab.
Sep 29 17:53:28 dist-test-slave-53nh krb5kdc[6478](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704408, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T17:53:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T17:53:28Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704375975685-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704375975685-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260929 17:53:28.388015 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskup3Soe/build/debug/bin/kudu
/tmp/dist-test-taskup3Soe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704375975685-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704375975685-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704375975685-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704375975685-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39863
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704375975685-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39863
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704375975685-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704375975685-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704375975685-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:53:28.494736 6491 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:53:28.495002 6491 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:53:28.495040 6491 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:53:28.499095 6491 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 17:53:28.499156 6491 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:53:28.499174 6491 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:53:28.499190 6491 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 17:53:28.499205 6491 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 17:53:28.505281 6491 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704375975685-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704375975685-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39863
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704375975685-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39863
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704375975685-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6491
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704375975685-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:53:28.506398 6491 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:53:28.507381 6491 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:53:28.515405 6496 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:53:28.515538 6491 server_base.cc:1034] running on GCE node
W20260929 17:53:28.515372 6499 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:53:28.515388 6497 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:53:28.516141 6491 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:53:28.517205 6491 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:53:28.518361 6491 hybrid_clock.cc:648] HybridClock initialized: now 1790704408518346 us; error 28 us; skew 500 ppm
Sep 29 17:53:28 dist-test-slave-53nh krb5kdc[6478](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704408, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:53:28.520934 6491 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260929 17:53:28.521759 6491 webserver.cc:460] Webserver started at http://127.0.98.62:32945/ using document root <none> and password file <none>
I20260929 17:53:28.522262 6491 fs_manager.cc:359] Metadata directory not provided
I20260929 17:53:28.522324 6491 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:53:28.522521 6491 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:53:28.524468 6491 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704375975685-392-0/minicluster-data/master-0/data/instance:
uuid: "14cabac4c134442b901728b1c630deb5"
format_stamp: "Formatted at 2026-09-29 17:53:28 on dist-test-slave-53nh"
server_key: "7c26f32aa6f0c95f2001e865788240bf"
server_key_iv: "4997df1da22e5e9552db5fcac251ff73"
server_key_version: "encryptionkey@0"
I20260929 17:53:28.524988 6491 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704375975685-392-0/minicluster-data/master-0/wal/instance:
uuid: "14cabac4c134442b901728b1c630deb5"
format_stamp: "Formatted at 2026-09-29 17:53:28 on dist-test-slave-53nh"
server_key: "7c26f32aa6f0c95f2001e865788240bf"
server_key_iv: "4997df1da22e5e9552db5fcac251ff73"
server_key_version: "encryptionkey@0"
I20260929 17:53:28.528434 6491 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260929 17:53:28.530678 6506 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:53:28.531821 6491 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 17:53:28.531934 6491 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704375975685-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704375975685-392-0/minicluster-data/master-0/wal
uuid: "14cabac4c134442b901728b1c630deb5"
format_stamp: "Formatted at 2026-09-29 17:53:28 on dist-test-slave-53nh"
server_key: "7c26f32aa6f0c95f2001e865788240bf"
server_key_iv: "4997df1da22e5e9552db5fcac251ff73"
server_key_version: "encryptionkey@0"
I20260929 17:53:28.532018 6491 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704375975685-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704375975685-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704375975685-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:53:28.547989 6491 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:53:28.550691 6491 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:53:28.550870 6491 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:53:28.557118 6491 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39863
I20260929 17:53:28.557372 6558 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39863 every 8 connection(s)
I20260929 17:53:28.558019 6491 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704375975685-392-0/minicluster-data/master-0/data/info.pb
I20260929 17:53:28.560798 6559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:53:28.563591 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskup3Soe/build/debug/bin/kudu as pid 6491
I20260929 17:53:28.563717 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704375975685-392-0/minicluster-data/master-0/wal/instance
I20260929 17:53:28.564011 392 external_mini_cluster.cc:1428] Setting key 560cd9008cdae3750a2bc24f52a86a95
I20260929 17:53:28.566636 6559 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14cabac4c134442b901728b1c630deb5: Bootstrap starting.
I20260929 17:53:28.568737 6559 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 14cabac4c134442b901728b1c630deb5: Neither blocks nor log segments found. Creating new log.
I20260929 17:53:28.569489 6559 log.cc:824] T 00000000000000000000000000000000 P 14cabac4c134442b901728b1c630deb5: Log is configured to *not* fsync() on all Append() calls
I20260929 17:53:28.571261 6559 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14cabac4c134442b901728b1c630deb5: No bootstrap required, opened a new log
Sep 29 17:53:28 dist-test-slave-53nh krb5kdc[6478](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704408, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260929 17:53:28.574735 6559 raft_consensus.cc:348] T 00000000000000000000000000000000 P 14cabac4c134442b901728b1c630deb5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14cabac4c134442b901728b1c630deb5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39863 } }
I20260929 17:53:28.574987 6559 raft_consensus.cc:374] T 00000000000000000000000000000000 P 14cabac4c134442b901728b1c630deb5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:53:28.575045 6559 raft_consensus.cc:729] T 00000000000000000000000000000000 P 14cabac4c134442b901728b1c630deb5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14cabac4c134442b901728b1c630deb5, State: Initialized, Role: FOLLOWER
I20260929 17:53:28.575484 6559 consensus_queue.cc:260] T 00000000000000000000000000000000 P 14cabac4c134442b901728b1c630deb5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14cabac4c134442b901728b1c630deb5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39863 } }
I20260929 17:53:28.575623 6559 raft_consensus.cc:388] T 00000000000000000000000000000000 P 14cabac4c134442b901728b1c630deb5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 17:53:28.575687 6559 raft_consensus.cc:482] T 00000000000000000000000000000000 P 14cabac4c134442b901728b1c630deb5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 17:53:28.575789 6559 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 14cabac4c134442b901728b1c630deb5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:53:28.576625 6559 raft_consensus.cc:504] T 00000000000000000000000000000000 P 14cabac4c134442b901728b1c630deb5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14cabac4c134442b901728b1c630deb5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39863 } }
I20260929 17:53:28.576947 6559 leader_election.cc:302] T 00000000000000000000000000000000 P 14cabac4c134442b901728b1c630deb5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 14cabac4c134442b901728b1c630deb5; no voters:
I20260929 17:53:28.577301 6559 leader_election.cc:288] T 00000000000000000000000000000000 P 14cabac4c134442b901728b1c630deb5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 17:53:28.577358 6564 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 14cabac4c134442b901728b1c630deb5 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 17:53:28.577572 6564 raft_consensus.cc:686] T 00000000000000000000000000000000 P 14cabac4c134442b901728b1c630deb5 [term 1 LEADER]: Becoming Leader. State: Replica: 14cabac4c134442b901728b1c630deb5, State: Running, Role: LEADER
I20260929 17:53:28.577939 6564 consensus_queue.cc:237] T 00000000000000000000000000000000 P 14cabac4c134442b901728b1c630deb5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14cabac4c134442b901728b1c630deb5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39863 } }
I20260929 17:53:28.578480 6562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:28.565506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40034 (local address 127.0.98.62:39863)
0929 17:53:28.565920 (+ 414us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:28.565934 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:28.565971 (+ 37us) server_negotiation.cc:407] Connection header received
0929 17:53:28.566556 (+ 585us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:28.566572 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:28.566928 (+ 356us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:28.567227 (+ 299us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:53:28.567965 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:28.568971 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:28.569849 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:28.570316 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:28.570456 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:28.572981 (+ 2525us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:53:28.572986 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:53:28.573032 (+ 46us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:53:28.573062 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:53:28.575242 (+ 2180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:53:28.575647 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:53:28.575652 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:53:28.575657 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:53:28.575799 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:53:28.576063 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:53:28.576066 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:53:28.576068 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:53:28.576450 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:53:28.576619 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0929 17:53:28.577228 (+ 609us) server_negotiation.cc:299] Negotiation successful
0929 17:53:28.577549 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":120,"threads_started":1}
I20260929 17:53:28.578539 6559 sys_catalog.cc:564] T 00000000000000000000000000000000 P 14cabac4c134442b901728b1c630deb5 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 17:53:28.578998 6564 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14cabac4c134442b901728b1c630deb5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 14cabac4c134442b901728b1c630deb5. Latest consensus state: current_term: 1 leader_uuid: "14cabac4c134442b901728b1c630deb5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14cabac4c134442b901728b1c630deb5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39863 } } }
I20260929 17:53:28.579185 6564 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14cabac4c134442b901728b1c630deb5 [sys.catalog]: This master's current role is: LEADER
I20260929 17:53:28.579406 6565 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14cabac4c134442b901728b1c630deb5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "14cabac4c134442b901728b1c630deb5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14cabac4c134442b901728b1c630deb5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39863 } } }
I20260929 17:53:28.579542 6565 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14cabac4c134442b901728b1c630deb5 [sys.catalog]: This master's current role is: LEADER
I20260929 17:53:28.579593 6571 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 17:53:28.582389 6571 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 17:53:28.587220 6571 catalog_manager.cc:1347] Generated new cluster ID: 206c4021e40644f6bf3327638ec8f5f8
I20260929 17:53:28.587291 6571 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 17:53:28.617736 6571 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 17:53:28.618746 6571 catalog_manager.cc:1504] Loading token signing keys...
I20260929 17:53:28.627555 6571 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 14cabac4c134442b901728b1c630deb5: Generated new TSK 0
I20260929 17:53:28.628329 6571 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704375975685-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704375975685-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260929 17:53:28.656085 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskup3Soe/build/debug/bin/kudu
/tmp/dist-test-taskup3Soe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704375975685-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704375975685-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704375975685-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704375975685-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39863
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704375975685-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704375975685-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704375975685-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704375975685-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:53:28.762426 6585 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:53:28.762686 6585 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:53:28.762725 6585 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:53:28.766659 6585 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:53:28.766712 6585 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:53:28.766769 6585 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 17:53:28.771868 6585 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704375975685-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704375975685-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704375975685-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704375975685-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39863
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6585
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704375975685-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:53:28.772833 6585 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:53:28.773790 6585 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:53:28.782869 6590 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:53:28.782857 6593 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:53:28.783023 6585 server_base.cc:1034] running on GCE node
W20260929 17:53:28.782857 6591 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:53:28.783547 6585 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:53:28.784168 6585 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:53:28.785318 6585 hybrid_clock.cc:648] HybridClock initialized: now 1790704408785302 us; error 26 us; skew 500 ppm
Sep 29 17:53:28 dist-test-slave-53nh krb5kdc[6478](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704408, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:53:28.787919 6585 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260929 17:53:28.788718 6585 webserver.cc:460] Webserver started at http://127.0.98.1:41565/ using document root <none> and password file <none>
I20260929 17:53:28.789292 6585 fs_manager.cc:359] Metadata directory not provided
I20260929 17:53:28.789359 6585 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:53:28.789547 6585 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:53:28.791538 6585 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704375975685-392-0/minicluster-data/ts-0/data/instance:
uuid: "7cb83ff6a04043169a3cf2e232462596"
format_stamp: "Formatted at 2026-09-29 17:53:28 on dist-test-slave-53nh"
server_key: "3ac0901fb019f271eff2471049c6f14a"
server_key_iv: "94db964e8713f1ec1a0f1698e158718f"
server_key_version: "encryptionkey@0"
I20260929 17:53:28.792078 6585 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704375975685-392-0/minicluster-data/ts-0/wal/instance:
uuid: "7cb83ff6a04043169a3cf2e232462596"
format_stamp: "Formatted at 2026-09-29 17:53:28 on dist-test-slave-53nh"
server_key: "3ac0901fb019f271eff2471049c6f14a"
server_key_iv: "94db964e8713f1ec1a0f1698e158718f"
server_key_version: "encryptionkey@0"
I20260929 17:53:28.795594 6585 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260929 17:53:28.797950 6600 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:53:28.799187 6585 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 17:53:28.799297 6585 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704375975685-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704375975685-392-0/minicluster-data/ts-0/wal
uuid: "7cb83ff6a04043169a3cf2e232462596"
format_stamp: "Formatted at 2026-09-29 17:53:28 on dist-test-slave-53nh"
server_key: "3ac0901fb019f271eff2471049c6f14a"
server_key_iv: "94db964e8713f1ec1a0f1698e158718f"
server_key_version: "encryptionkey@0"
I20260929 17:53:28.799393 6585 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704375975685-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704375975685-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704375975685-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:53:28.815546 6585 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:53:28.818601 6585 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:53:28.818775 6585 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:53:28.819322 6585 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 17:53:28.820086 6585 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:53:28.820132 6585 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:53:28.820184 6585 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:53:28.820210 6585 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 17:53:28.828265 6585 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46193
I20260929 17:53:28.828280 6713 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46193 every 8 connection(s)
I20260929 17:53:28.829370 6585 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704375975685-392-0/minicluster-data/ts-0/data/info.pb
I20260929 17:53:28.831601 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskup3Soe/build/debug/bin/kudu as pid 6585
I20260929 17:53:28.831709 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704375975685-392-0/minicluster-data/ts-0/wal/instance
I20260929 17:53:28.832019 392 external_mini_cluster.cc:1428] Setting key 10eaba359a33d85bc5d86d3a63ecdb60
Sep 29 17:53:28 dist-test-slave-53nh krb5kdc[6478](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704408, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
I20260929 17:53:28.843921 6562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:28.831130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36491 (local address 127.0.98.62:39863)
0929 17:53:28.831272 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:28.831276 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:28.832175 (+ 899us) server_negotiation.cc:407] Connection header received
0929 17:53:28.833732 (+ 1557us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:28.833735 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:28.833777 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:28.833854 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:53:28.835398 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:28.836218 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:28.837440 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:28.837780 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:28.837895 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:28.840435 (+ 2540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:53:28.840438 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:53:28.840440 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:53:28.840462 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:53:28.841822 (+ 1360us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:53:28.842345 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:53:28.842348 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:53:28.842349 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:53:28.842426 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:53:28.842755 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:53:28.842758 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:53:28.842760 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:53:28.842942 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:53:28.843029 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0929 17:53:28.843558 (+ 529us) server_negotiation.cc:299] Negotiation successful
0929 17:53:28.843738 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 17:53:28.844910 6716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:28.831532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39863 (local address 127.0.98.1:36491)
0929 17:53:28.832003 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:28.832061 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0929 17:53:28.833518 (+ 1457us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:53:28.834044 (+ 526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:53:28.834053 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:53:28.834541 (+ 488us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:53:28.835240 (+ 699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:28.835260 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:28.836329 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:28.836332 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:28.837324 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:28.837334 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:28.837930 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:28.837933 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:28.838081 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:28.838701 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:53:28.838717 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:53:28.840254 (+ 1537us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:53:28.841923 (+ 1669us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:53:28.841928 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:53:28.841942 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:53:28.842232 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:53:28.842510 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:53:28.842513 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:53:28.842515 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:53:28.842666 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:53:28.843041 (+ 375us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:53:28.843048 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:53:28.843326 (+ 278us) client_negotiation.cc:769] Sending connection context
0929 17:53:28.843506 (+ 180us) client_negotiation.cc:240] Negotiation successful
0929 17:53:28.843874 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":103,"threads_started":1}
I20260929 17:53:28.846246 6714 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39863
I20260929 17:53:28.846527 6714 heartbeater.cc:464] Registering TS with master...
I20260929 17:53:28.847107 6714 heartbeater.cc:511] Master 127.0.98.62:39863 requested a full tablet report, sending...
I20260929 17:53:28.848889 6523 ts_manager.cc:194] Registered new tserver with Master: 7cb83ff6a04043169a3cf2e232462596 (127.0.98.1:46193)
I20260929 17:53:28.850539 6523 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:36491
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704375975685-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704375975685-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260929 17:53:28.859913 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskup3Soe/build/debug/bin/kudu
/tmp/dist-test-taskup3Soe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704375975685-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704375975685-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704375975685-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704375975685-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39863
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704375975685-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704375975685-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704375975685-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704375975685-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:53:28.965912 6719 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:53:28.966178 6719 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:53:28.966224 6719 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:53:28.970235 6719 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:53:28.970300 6719 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:53:28.970359 6719 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 17:53:28.976032 6719 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704375975685-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704375975685-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704375975685-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704375975685-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39863
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6719
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704375975685-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:53:28.977154 6719 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:53:28.978166 6719 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:53:28.987537 6725 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:53:28.987694 6724 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:53:28.987555 6727 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:53:28.987840 6719 server_base.cc:1034] running on GCE node
I20260929 17:53:28.988288 6719 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:53:28.988921 6719 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:53:28.990079 6719 hybrid_clock.cc:648] HybridClock initialized: now 1790704408990059 us; error 36 us; skew 500 ppm
Sep 29 17:53:28 dist-test-slave-53nh krb5kdc[6478](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704408, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:53:28.992635 6719 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260929 17:53:28.993535 6719 webserver.cc:460] Webserver started at http://127.0.98.2:33415/ using document root <none> and password file <none>
I20260929 17:53:28.994056 6719 fs_manager.cc:359] Metadata directory not provided
I20260929 17:53:28.994118 6719 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:53:28.994320 6719 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:53:28.996327 6719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704375975685-392-0/minicluster-data/ts-1/data/instance:
uuid: "ade19fb6904045318b25a43744e7c3f3"
format_stamp: "Formatted at 2026-09-29 17:53:28 on dist-test-slave-53nh"
server_key: "fc81c3bac5f6f6f84b78f8fbd77d8000"
server_key_iv: "1c42630f4150badf894fe6361a951c14"
server_key_version: "encryptionkey@0"
I20260929 17:53:28.996851 6719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704375975685-392-0/minicluster-data/ts-1/wal/instance:
uuid: "ade19fb6904045318b25a43744e7c3f3"
format_stamp: "Formatted at 2026-09-29 17:53:28 on dist-test-slave-53nh"
server_key: "fc81c3bac5f6f6f84b78f8fbd77d8000"
server_key_iv: "1c42630f4150badf894fe6361a951c14"
server_key_version: "encryptionkey@0"
I20260929 17:53:29.000502 6719 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260929 17:53:29.003051 6734 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:53:29.004371 6719 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 17:53:29.004503 6719 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704375975685-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704375975685-392-0/minicluster-data/ts-1/wal
uuid: "ade19fb6904045318b25a43744e7c3f3"
format_stamp: "Formatted at 2026-09-29 17:53:28 on dist-test-slave-53nh"
server_key: "fc81c3bac5f6f6f84b78f8fbd77d8000"
server_key_iv: "1c42630f4150badf894fe6361a951c14"
server_key_version: "encryptionkey@0"
I20260929 17:53:29.004630 6719 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704375975685-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704375975685-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704375975685-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:53:29.025512 6719 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:53:29.028894 6719 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:53:29.029094 6719 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:53:29.029670 6719 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 17:53:29.030463 6719 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:53:29.030508 6719 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:53:29.030561 6719 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:53:29.030589 6719 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 17:53:29.039083 6719 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43951
I20260929 17:53:29.039110 6847 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43951 every 8 connection(s)
I20260929 17:53:29.039939 6719 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704375975685-392-0/minicluster-data/ts-1/data/info.pb
I20260929 17:53:29.045936 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskup3Soe/build/debug/bin/kudu as pid 6719
I20260929 17:53:29.046048 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704375975685-392-0/minicluster-data/ts-1/wal/instance
I20260929 17:53:29.046310 392 external_mini_cluster.cc:1428] Setting key d6abe990efdcdcd26152d2d1fd57aa2a
Sep 29 17:53:29 dist-test-slave-53nh krb5kdc[6478](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704408, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260929 17:53:29.053529 6562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:29.041714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41975 (local address 127.0.98.62:39863)
0929 17:53:29.041882 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:29.041887 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:29.042702 (+ 815us) server_negotiation.cc:407] Connection header received
0929 17:53:29.043966 (+ 1264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:29.043968 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:29.044014 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:29.044086 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:53:29.045237 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:29.045831 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:29.047035 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:29.047329 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:29.047430 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:29.049933 (+ 2503us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:53:29.049936 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:53:29.049938 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:53:29.049960 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:53:29.051384 (+ 1424us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:53:29.051877 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:53:29.051881 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:53:29.051883 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:53:29.051976 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:53:29.052361 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:53:29.052364 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:53:29.052365 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:53:29.052544 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:53:29.052617 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0929 17:53:29.053305 (+ 688us) server_negotiation.cc:299] Negotiation successful
0929 17:53:29.053410 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 17:53:29.054642 6850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:29.042124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39863 (local address 127.0.98.2:41975)
0929 17:53:29.042556 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:29.042599 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 17:53:29.043770 (+ 1171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:53:29.044237 (+ 467us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:53:29.044244 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:53:29.044588 (+ 344us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:53:29.045116 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:29.045131 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:29.045932 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:29.045934 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:29.046927 (+ 993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:29.046937 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:29.047463 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:29.047467 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:29.047623 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:29.048165 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:53:29.048181 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:53:29.049778 (+ 1597us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:53:29.051487 (+ 1709us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:53:29.051493 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:53:29.051508 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:53:29.051784 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:53:29.052072 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:53:29.052075 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:53:29.052077 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:53:29.052263 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:53:29.052637 (+ 374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:53:29.052666 (+ 29us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:53:29.052988 (+ 322us) client_negotiation.cc:769] Sending connection context
0929 17:53:29.053260 (+ 272us) client_negotiation.cc:240] Negotiation successful
0929 17:53:29.053601 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":113,"threads_started":1}
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260929 17:53:29.055748 6848 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39863
I20260929 17:53:29.055987 6848 heartbeater.cc:464] Registering TS with master...
I20260929 17:53:29.056444 6848 heartbeater.cc:511] Master 127.0.98.62:39863 requested a full tablet report, sending...
I20260929 17:53:29.057456 6523 ts_manager.cc:194] Registered new tserver with Master: ade19fb6904045318b25a43744e7c3f3 (127.0.98.2:43951)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
I20260929 17:53:29.058068 6523 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:41975
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704375975685-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704375975685-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260929 17:53:29.071920 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskup3Soe/build/debug/bin/kudu
/tmp/dist-test-taskup3Soe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704375975685-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704375975685-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704375975685-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704375975685-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39863
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704375975685-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704375975685-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704375975685-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704375975685-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:53:29.179565 6853 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:53:29.179832 6853 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:53:29.179874 6853 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:53:29.183868 6853 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:53:29.183925 6853 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:53:29.183986 6853 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 17:53:29.189313 6853 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704375975685-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704375975685-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704375975685-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704375975685-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39863
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6853
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704375975685-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:53:29.190366 6853 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:53:29.191358 6853 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:53:29.200486 6861 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:53:29.200572 6853 server_base.cc:1034] running on GCE node
W20260929 17:53:29.200472 6858 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:53:29.200472 6859 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:53:29.201159 6853 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:53:29.201776 6853 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:53:29.202922 6853 hybrid_clock.cc:648] HybridClock initialized: now 1790704409202897 us; error 37 us; skew 500 ppm
Sep 29 17:53:29 dist-test-slave-53nh krb5kdc[6478](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704409, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:53:29.205583 6853 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260929 17:53:29.206399 6853 webserver.cc:460] Webserver started at http://127.0.98.3:43769/ using document root <none> and password file <none>
I20260929 17:53:29.206930 6853 fs_manager.cc:359] Metadata directory not provided
I20260929 17:53:29.207005 6853 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:53:29.207208 6853 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:53:29.209277 6853 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704375975685-392-0/minicluster-data/ts-2/data/instance:
uuid: "c4d0d8b2d8994344a6ec327d5c389bba"
format_stamp: "Formatted at 2026-09-29 17:53:29 on dist-test-slave-53nh"
server_key: "d1ad7677298cd0e0e27082c0e2868c8a"
server_key_iv: "156845e626864f41fc4bc5d0920d7255"
server_key_version: "encryptionkey@0"
I20260929 17:53:29.209766 6853 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704375975685-392-0/minicluster-data/ts-2/wal/instance:
uuid: "c4d0d8b2d8994344a6ec327d5c389bba"
format_stamp: "Formatted at 2026-09-29 17:53:29 on dist-test-slave-53nh"
server_key: "d1ad7677298cd0e0e27082c0e2868c8a"
server_key_iv: "156845e626864f41fc4bc5d0920d7255"
server_key_version: "encryptionkey@0"
I20260929 17:53:29.213265 6853 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260929 17:53:29.215540 6868 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:53:29.216737 6853 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 17:53:29.216832 6853 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704375975685-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704375975685-392-0/minicluster-data/ts-2/wal
uuid: "c4d0d8b2d8994344a6ec327d5c389bba"
format_stamp: "Formatted at 2026-09-29 17:53:29 on dist-test-slave-53nh"
server_key: "d1ad7677298cd0e0e27082c0e2868c8a"
server_key_iv: "156845e626864f41fc4bc5d0920d7255"
server_key_version: "encryptionkey@0"
I20260929 17:53:29.216928 6853 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704375975685-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704375975685-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704375975685-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:53:29.234411 6853 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:53:29.237757 6853 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:53:29.237914 6853 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:53:29.238476 6853 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 17:53:29.239275 6853 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:53:29.239321 6853 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:53:29.239359 6853 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:53:29.239377 6853 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 17:53:29.247495 6853 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41685
I20260929 17:53:29.247527 6981 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41685 every 8 connection(s)
I20260929 17:53:29.248387 6853 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704375975685-392-0/minicluster-data/ts-2/data/info.pb
Sep 29 17:53:29 dist-test-slave-53nh krb5kdc[6478](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704409, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260929 17:53:29.258003 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskup3Soe/build/debug/bin/kudu as pid 6853
I20260929 17:53:29.258111 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704375975685-392-0/minicluster-data/ts-2/wal/instance
I20260929 17:53:29.258409 392 external_mini_cluster.cc:1428] Setting key fb875c5d03a6facac85aa8eac8aca6a0
I20260929 17:53:29.262095 6562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:29.250270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41279 (local address 127.0.98.62:39863)
0929 17:53:29.250439 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:29.250443 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:29.251298 (+ 855us) server_negotiation.cc:407] Connection header received
0929 17:53:29.252666 (+ 1368us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:29.252670 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:29.252727 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:29.252800 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:53:29.254041 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:29.254576 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:29.255703 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:29.255920 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:29.256026 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:29.258652 (+ 2626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:53:29.258655 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:53:29.258657 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:53:29.258682 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:53:29.260230 (+ 1548us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:53:29.260775 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:53:29.260778 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:53:29.260779 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:53:29.260855 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:53:29.261173 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:53:29.261176 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:53:29.261177 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:53:29.261352 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:53:29.261423 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0929 17:53:29.261877 (+ 454us) server_negotiation.cc:299] Negotiation successful
0929 17:53:29.261976 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 17:53:29.262902 6984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:29.250635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39863 (local address 127.0.98.3:41279)
0929 17:53:29.251083 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:29.251171 (+ 88us) client_negotiation.cc:174] Beginning negotiation
0929 17:53:29.252429 (+ 1258us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:53:29.252985 (+ 556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:53:29.252992 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:53:29.253404 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:53:29.253891 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:29.253904 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:29.254699 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:29.254702 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:29.255585 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:29.255594 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:29.256064 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:29.256069 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:29.256238 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:29.256803 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:53:29.256819 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:53:29.258460 (+ 1641us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:53:29.260344 (+ 1884us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:53:29.260350 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:53:29.260367 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:53:29.260696 (+ 329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:53:29.260925 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:53:29.260927 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:53:29.260929 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:53:29.261103 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:53:29.261424 (+ 321us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:53:29.261429 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:53:29.261667 (+ 238us) client_negotiation.cc:769] Sending connection context
0929 17:53:29.261833 (+ 166us) client_negotiation.cc:240] Negotiation successful
0929 17:53:29.262095 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":101,"threads_started":1}
I20260929 17:53:29.263940 6982 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39863
I20260929 17:53:29.264160 6982 heartbeater.cc:464] Registering TS with master...
I20260929 17:53:29.264622 6982 heartbeater.cc:511] Master 127.0.98.62:39863 requested a full tablet report, sending...
I20260929 17:53:29.265569 6523 ts_manager.cc:194] Registered new tserver with Master: c4d0d8b2d8994344a6ec327d5c389bba (127.0.98.3:41685)
I20260929 17:53:29.266115 6523 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:41279
I20260929 17:53:29.272411 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704375975685-392-0/krb5kdc/oryx_127.0.98.61.keytab.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704375975685-392-0/krb5kdc/oryx_127.0.98.61.keytab.
I20260929 17:53:29.298373 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskup3Soe/build/debug/bin/kudu
/tmp/dist-test-taskup3Soe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704375975685-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704375975685-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704375975685-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704375975685-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:34183
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704375975685-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39863,127.0.98.61:34183
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704375975685-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704375975685-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704375975685-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:53:29.407210 6987 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:53:29.407517 6987 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:53:29.407567 6987 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:53:29.411772 6987 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 17:53:29.411844 6987 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:53:29.411862 6987 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:53:29.411877 6987 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 17:53:29.411891 6987 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 17:53:29.417316 6987 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704375975685-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704375975685-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39863,127.0.98.61:34183
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704375975685-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:34183
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704375975685-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6987
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704375975685-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
W20260929 17:53:29.417548 6987 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.0.98.62:39863,127.0.98.61:34183'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260929 17:53:29.418434 6987 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:53:29.419435 6987 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:53:29.427677 6993 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:53:29.427706 6987 server_base.cc:1034] running on GCE node
W20260929 17:53:29.427709 6992 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:53:29.427737 6995 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:53:29.428442 6987 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:53:29.429620 6987 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:53:29.430773 6987 hybrid_clock.cc:648] HybridClock initialized: now 1790704409430754 us; error 32 us; skew 500 ppm
Sep 29 17:53:29 dist-test-slave-53nh krb5kdc[6478](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704409, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:53:29.433386 6987 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260929 17:53:29.434271 6987 webserver.cc:460] Webserver started at http://127.0.98.61:41495/ using document root <none> and password file <none>
I20260929 17:53:29.434806 6987 fs_manager.cc:359] Metadata directory not provided
I20260929 17:53:29.434861 6987 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:53:29.435029 6987 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:53:29.437151 6987 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704375975685-392-0/minicluster-data/master-1/data/instance:
uuid: "952a9b1a252747a8acdcfa8e04debe30"
format_stamp: "Formatted at 2026-09-29 17:53:29 on dist-test-slave-53nh"
server_key: "9a5cdd0c7b066ae9a56f951ebddcd967"
server_key_iv: "19f0f12433f729d216cab254efb3b8dd"
server_key_version: "encryptionkey@0"
I20260929 17:53:29.437664 6987 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704375975685-392-0/minicluster-data/master-1/wal/instance:
uuid: "952a9b1a252747a8acdcfa8e04debe30"
format_stamp: "Formatted at 2026-09-29 17:53:29 on dist-test-slave-53nh"
server_key: "9a5cdd0c7b066ae9a56f951ebddcd967"
server_key_iv: "19f0f12433f729d216cab254efb3b8dd"
server_key_version: "encryptionkey@0"
I20260929 17:53:29.441224 6987 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260929 17:53:29.443464 7002 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:53:29.444618 6987 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 17:53:29.444721 6987 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704375975685-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704375975685-392-0/minicluster-data/master-1/wal
uuid: "952a9b1a252747a8acdcfa8e04debe30"
format_stamp: "Formatted at 2026-09-29 17:53:29 on dist-test-slave-53nh"
server_key: "9a5cdd0c7b066ae9a56f951ebddcd967"
server_key_iv: "19f0f12433f729d216cab254efb3b8dd"
server_key_version: "encryptionkey@0"
I20260929 17:53:29.444828 6987 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704375975685-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704375975685-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704375975685-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:53:29.469056 6987 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:53:29.471948 6987 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:53:29.472123 6987 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:53:29.478974 6987 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:34183
I20260929 17:53:29.479225 7054 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:34183 every 8 connection(s)
I20260929 17:53:29.479950 6987 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704375975685-392-0/minicluster-data/master-1/data/info.pb
I20260929 17:53:29.482698 7055 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:53:29.484323 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskup3Soe/build/debug/bin/kudu as pid 6987
I20260929 17:53:29.486756 7055 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39863 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 17:53:29 dist-test-slave-53nh krb5kdc[6478](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704408, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Sep 29 17:53:29 dist-test-slave-53nh krb5kdc[6478](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704409, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260929 17:53:29.498219 6562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:29.488600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40042 (local address 127.0.98.62:39863)
0929 17:53:29.488730 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:29.488735 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:29.489040 (+ 305us) server_negotiation.cc:407] Connection header received
0929 17:53:29.489181 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:29.489184 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:29.489232 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:29.489301 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:53:29.490014 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:29.490834 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:29.492037 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:29.492324 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:29.492426 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:29.494757 (+ 2331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:53:29.494761 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:53:29.494764 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:53:29.494791 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:53:29.496105 (+ 1314us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:53:29.496827 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:53:29.496830 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:53:29.496831 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:53:29.496916 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:53:29.497182 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:53:29.497186 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:53:29.497187 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:53:29.497473 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:53:29.497556 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0929 17:53:29.497938 (+ 382us) server_negotiation.cc:299] Negotiation successful
0929 17:53:29.498054 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 17:53:29.498968 7058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:29.485802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41434 (local address 127.0.98.61:34183)
0929 17:53:29.486242 (+ 440us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:29.486256 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:29.486311 (+ 55us) server_negotiation.cc:407] Connection header received
0929 17:53:29.486948 (+ 637us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:29.486967 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:29.487323 (+ 356us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:29.487654 (+ 331us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:53:29.488424 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:29.489897 (+ 1473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:29.491037 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:29.491740 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:29.491900 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:29.494254 (+ 2354us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:53:29.494260 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:53:29.494271 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:53:29.494292 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:53:29.496096 (+ 1804us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:53:29.496480 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:53:29.496485 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:53:29.496490 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:53:29.496586 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:53:29.496874 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:53:29.496877 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:53:29.496879 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:53:29.497373 (+ 494us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:53:29.497565 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0929 17:53:29.497802 (+ 237us) server_negotiation.cc:299] Negotiation successful
0929 17:53:29.498083 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":137,"threads_started":1}
I20260929 17:53:29.498970 7059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:29.488526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39863 (local address 127.0.0.1:40042)
0929 17:53:29.488898 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:29.488946 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0929 17:53:29.489092 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:53:29.489357 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:53:29.489363 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:53:29.489768 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:53:29.489914 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:29.489920 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:29.490945 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:29.490947 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:29.491924 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:29.491932 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:29.492469 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:29.492472 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:29.492546 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:29.493026 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:53:29.493049 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:53:29.494620 (+ 1571us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:53:29.496495 (+ 1875us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:53:29.496502 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:53:29.496506 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:53:29.496740 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:53:29.497018 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:53:29.497021 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:53:29.497023 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:53:29.497103 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:53:29.497563 (+ 460us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:53:29.497570 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:53:29.497717 (+ 147us) client_negotiation.cc:769] Sending connection context
0929 17:53:29.497858 (+ 141us) client_negotiation.cc:240] Negotiation successful
0929 17:53:29.498176 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":199,"mutex_wait_us":74,"thread_start_us":105,"threads_started":1}
I20260929 17:53:29.501202 7055 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34183 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 17:53:29 dist-test-slave-53nh krb5kdc[6478](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704409, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260929 17:53:29.509357 7059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:29.501640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34183 (local address 127.0.0.1:41444)
0929 17:53:29.501736 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:29.501748 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 17:53:29.501809 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:53:29.501977 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:53:29.501979 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:53:29.502096 (+ 117us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:53:29.502224 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:29.502229 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:29.503508 (+ 1279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:29.503512 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:29.504272 (+ 760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:29.504281 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:29.504864 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:29.504868 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:29.504967 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:29.505443 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:53:29.505455 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:53:29.506729 (+ 1274us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:53:29.508161 (+ 1432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:53:29.508164 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:53:29.508165 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:53:29.508339 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:53:29.508523 (+ 184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:53:29.508525 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:53:29.508526 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:53:29.508584 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:53:29.508894 (+ 310us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:53:29.508896 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:53:29.508963 (+ 67us) client_negotiation.cc:769] Sending connection context
0929 17:53:29.509087 (+ 124us) client_negotiation.cc:240] Negotiation successful
0929 17:53:29.509204 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26}
I20260929 17:53:29.509449 7058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:29.501680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41444 (local address 127.0.98.61:34183)
0929 17:53:29.501776 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:29.501791 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:29.501802 (+ 11us) server_negotiation.cc:407] Connection header received
0929 17:53:29.501869 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:29.501871 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:29.501903 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:29.501960 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:53:29.502311 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:29.503397 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:29.504400 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:29.504749 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:29.504859 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:29.506861 (+ 2002us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:53:29.506864 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:53:29.506866 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:53:29.506885 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:53:29.508082 (+ 1197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:53:29.508409 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:53:29.508411 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:53:29.508412 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:53:29.508474 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:53:29.508636 (+ 162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:53:29.508638 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:53:29.508639 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:53:29.508825 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:53:29.508902 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0929 17:53:29.509156 (+ 254us) server_negotiation.cc:299] Negotiation successful
0929 17:53:29.509260 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260929 17:53:29.512135 7055 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 952a9b1a252747a8acdcfa8e04debe30: Bootstrap starting.
I20260929 17:53:29.514403 7055 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 952a9b1a252747a8acdcfa8e04debe30: Neither blocks nor log segments found. Creating new log.
I20260929 17:53:29.515095 7055 log.cc:824] T 00000000000000000000000000000000 P 952a9b1a252747a8acdcfa8e04debe30: Log is configured to *not* fsync() on all Append() calls
I20260929 17:53:29.516858 7055 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 952a9b1a252747a8acdcfa8e04debe30: No bootstrap required, opened a new log
I20260929 17:53:29.519750 7055 raft_consensus.cc:348] T 00000000000000000000000000000000 P 952a9b1a252747a8acdcfa8e04debe30 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14cabac4c134442b901728b1c630deb5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39863 } } peers { permanent_uuid: "952a9b1a252747a8acdcfa8e04debe30" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34183 } }
I20260929 17:53:29.519943 7055 raft_consensus.cc:374] T 00000000000000000000000000000000 P 952a9b1a252747a8acdcfa8e04debe30 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:53:29.519995 7055 raft_consensus.cc:729] T 00000000000000000000000000000000 P 952a9b1a252747a8acdcfa8e04debe30 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 952a9b1a252747a8acdcfa8e04debe30, State: Initialized, Role: FOLLOWER
I20260929 17:53:29.520397 7055 consensus_queue.cc:260] T 00000000000000000000000000000000 P 952a9b1a252747a8acdcfa8e04debe30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14cabac4c134442b901728b1c630deb5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39863 } } peers { permanent_uuid: "952a9b1a252747a8acdcfa8e04debe30" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34183 } }
I20260929 17:53:29.520831 7061 sys_catalog.cc:455] T 00000000000000000000000000000000 P 952a9b1a252747a8acdcfa8e04debe30 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14cabac4c134442b901728b1c630deb5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39863 } } peers { permanent_uuid: "952a9b1a252747a8acdcfa8e04debe30" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34183 } } }
I20260929 17:53:29.520946 7061 sys_catalog.cc:458] T 00000000000000000000000000000000 P 952a9b1a252747a8acdcfa8e04debe30 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 17:53:29.521557 7055 sys_catalog.cc:564] T 00000000000000000000000000000000 P 952a9b1a252747a8acdcfa8e04debe30 [sys.catalog]: configured and running, proceeding with master startup.
W20260929 17:53:29.525808 7072 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 952a9b1a252747a8acdcfa8e04debe30: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 17:53:29.525888 7072 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 17:53:29.527245 6987 master_runner.cc:428] Detected that this master 952a9b1a252747a8acdcfa8e04debe30 is joining an existing cluster
I20260929 17:53:29.527292 6987 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.61:34183
I20260929 17:53:29.536731 7058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:29.529111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41446 (local address 127.0.98.61:34183)
0929 17:53:29.529216 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:29.529220 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:29.529660 (+ 440us) server_negotiation.cc:407] Connection header received
0929 17:53:29.529795 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:29.529798 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:29.529833 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:29.529877 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:53:29.530430 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:29.531064 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:29.531813 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:29.532051 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:29.532136 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:29.533412 (+ 1276us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:53:29.533415 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:53:29.533418 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:53:29.533435 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:53:29.535063 (+ 1628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:53:29.535627 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:53:29.535630 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:53:29.535631 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:53:29.535701 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:53:29.535955 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:53:29.535958 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:53:29.535960 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:53:29.536110 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:53:29.536187 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0929 17:53:29.536476 (+ 289us) server_negotiation.cc:299] Negotiation successful
0929 17:53:29.536578 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260929 17:53:29.537257 6562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:29.529398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40052 (local address 127.0.98.62:39863)
0929 17:53:29.529525 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:29.529529 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:29.529542 (+ 13us) server_negotiation.cc:407] Connection header received
0929 17:53:29.529617 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:29.529620 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:29.529661 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:29.529726 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:53:29.530132 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:29.530841 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:29.532013 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:29.532313 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:29.532396 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:29.533394 (+ 998us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:53:29.533398 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:53:29.533401 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:53:29.533425 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:53:29.535160 (+ 1735us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:53:29.535591 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:53:29.535594 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:53:29.535596 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:53:29.535680 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:53:29.536236 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:53:29.536240 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:53:29.536241 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:53:29.536434 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:53:29.536538 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0929 17:53:29.536959 (+ 421us) server_negotiation.cc:299] Negotiation successful
0929 17:53:29.537109 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 17:53:29.537601 7078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:29.529215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34183 (local address 127.0.0.1:41446)
0929 17:53:29.529588 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:29.529602 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 17:53:29.529699 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:53:29.529991 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:53:29.529993 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:53:29.530199 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:53:29.530324 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:29.530331 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:29.531151 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:29.531154 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:29.531719 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:29.531726 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:29.532155 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:29.532158 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:29.532260 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:29.532771 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:53:29.532786 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:53:29.533275 (+ 489us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:53:29.535169 (+ 1894us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:53:29.535173 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:53:29.535175 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:53:29.535465 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:53:29.535787 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:53:29.535790 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:53:29.535791 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:53:29.535879 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:53:29.536190 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:53:29.536193 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:53:29.536322 (+ 129us) client_negotiation.cc:769] Sending connection context
0929 17:53:29.536858 (+ 536us) client_negotiation.cc:240] Negotiation successful
0929 17:53:29.537458 (+ 600us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"mutex_wait_us":445,"thread_start_us":75,"threads_started":1}
I20260929 17:53:29.537719 7079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:29.529067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39863 (local address 127.0.0.1:40052)
0929 17:53:29.529444 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:29.529458 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 17:53:29.529542 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:53:29.529747 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:53:29.529750 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:53:29.529938 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:53:29.530027 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:29.530032 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:29.530943 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:29.530946 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:29.531893 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:29.531902 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:29.532403 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:29.532407 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:29.532498 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:29.532895 (+ 397us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:53:29.532910 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:53:29.533281 (+ 371us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:53:29.535239 (+ 1958us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:53:29.535242 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:53:29.535244 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:53:29.535512 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:53:29.535765 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:53:29.535768 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:53:29.535770 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:53:29.535870 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:53:29.536539 (+ 669us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:53:29.536543 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:53:29.536642 (+ 99us) client_negotiation.cc:769] Sending connection context
0929 17:53:29.536745 (+ 103us) client_negotiation.cc:240] Negotiation successful
0929 17:53:29.537149 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":94,"threads_started":1}
Sep 29 17:53:29 dist-test-slave-53nh krb5kdc[6478](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704408, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260929 17:53:29.550160 7058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:29.541815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41448 (local address 127.0.98.61:34183)
0929 17:53:29.541918 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:29.541923 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:29.542204 (+ 281us) server_negotiation.cc:407] Connection header received
0929 17:53:29.542397 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:29.542400 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:29.542449 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:29.542519 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:53:29.543081 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:29.543853 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:29.544685 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:29.545038 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:29.545137 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:29.547291 (+ 2154us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:53:29.547294 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:53:29.547297 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:53:29.547316 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:53:29.548535 (+ 1219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:53:29.549059 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:53:29.549062 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:53:29.549064 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:53:29.549144 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:53:29.549399 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:53:29.549401 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:53:29.549402 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:53:29.549560 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:53:29.549628 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0929 17:53:29.549962 (+ 334us) server_negotiation.cc:299] Negotiation successful
0929 17:53:29.550044 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260929 17:53:29.550310 7080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:29.541771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34183 (local address 127.0.0.1:41448)
0929 17:53:29.542087 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:29.542136 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 17:53:29.542264 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:53:29.542551 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:53:29.542556 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:53:29.542889 (+ 333us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:53:29.542988 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:29.542993 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:29.543960 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:29.543963 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:29.544593 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:29.544601 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:29.545133 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:29.545136 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:29.545214 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:29.545689 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:53:29.545729 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:53:29.547150 (+ 1421us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:53:29.548627 (+ 1477us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:53:29.548636 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:53:29.548642 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:53:29.548937 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:53:29.549221 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:53:29.549224 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:53:29.549226 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:53:29.549325 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:53:29.549645 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:53:29.549651 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:53:29.549778 (+ 127us) client_negotiation.cc:769] Sending connection context
0929 17:53:29.549930 (+ 152us) client_negotiation.cc:240] Negotiation successful
0929 17:53:29.550125 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":84,"threads_started":1}
I20260929 17:53:29.551760 6523 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.61:34183: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "952a9b1a252747a8acdcfa8e04debe30"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.61"
port: 34183
}
attrs {
promote: true
}
}
dest_uuid: "14cabac4c134442b901728b1c630deb5"
cas_config_opid_index: -1
I20260929 17:53:29.552475 6523 consensus_queue.cc:237] T 00000000000000000000000000000000 P 14cabac4c134442b901728b1c630deb5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "14cabac4c134442b901728b1c630deb5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39863 } } peers { permanent_uuid: "952a9b1a252747a8acdcfa8e04debe30" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 34183 } attrs { promote: true } }
I20260929 17:53:29.554198 7083 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 14cabac4c134442b901728b1c630deb5 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 952a9b1a252747a8acdcfa8e04debe30 (127.0.98.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "14cabac4c134442b901728b1c630deb5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39863 } } peers { permanent_uuid: "952a9b1a252747a8acdcfa8e04debe30" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 34183 } attrs { promote: true } } }
I20260929 17:53:29.554986 7029 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 952a9b1a252747a8acdcfa8e04debe30 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:53:29.556130 7029 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 952a9b1a252747a8acdcfa8e04debe30 [term 1 FOLLOWER]: Refusing update from remote peer 14cabac4c134442b901728b1c630deb5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260929 17:53:29.556161 7083 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.61:34183
I20260929 17:53:29.556545 7082 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14cabac4c134442b901728b1c630deb5 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "14cabac4c134442b901728b1c630deb5" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "14cabac4c134442b901728b1c630deb5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39863 } } peers { permanent_uuid: "952a9b1a252747a8acdcfa8e04debe30" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 34183 } attrs { promote: true } } }
I20260929 17:53:29.556640 7082 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14cabac4c134442b901728b1c630deb5 [sys.catalog]: This master's current role is: LEADER
I20260929 17:53:29.556737 6987 master.cc:537] Master@127.0.98.61:34183 shutting down...
I20260929 17:53:29.556779 7083 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 14cabac4c134442b901728b1c630deb5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "952a9b1a252747a8acdcfa8e04debe30" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 34183 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260929 17:53:29.558583 6507 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260929 17:53:29.559314 6507 consensus_peers.cc:487] T 00000000000000000000000000000000 P 14cabac4c134442b901728b1c630deb5 -> Peer 952a9b1a252747a8acdcfa8e04debe30 (127.0.98.61:34183): Couldn't send request to peer 952a9b1a252747a8acdcfa8e04debe30. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
WARNING: no policy specified for oryx/127.0.98.60@KRBTEST.COM; defaulting to no policy
I20260929 17:53:29.564388 6987 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 952a9b1a252747a8acdcfa8e04debe30 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 17:53:29.564666 6987 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 952a9b1a252747a8acdcfa8e04debe30 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 17:53:29.564765 6987 tablet_replica.cc:330] T 00000000000000000000000000000000 P 952a9b1a252747a8acdcfa8e04debe30: stopping tablet replica
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.60@KRBTEST.COM" created.
I20260929 17:53:29.570155 6987 master.cc:552] Master@127.0.98.61:34183 shutdown complete.
I20260929 17:53:29.570267 6987 master_runner.cc:305] Clearing existing system tablet
I20260929 17:53:29.570561 6987 fs_manager.cc:359] Metadata directory not provided
I20260929 17:53:29.570668 6987 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:53:29.572000 6987 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 17:53:29.572881 7088 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:53:29.573154 6987 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704375975685-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:53:29.573267 6987 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 17:53:29.573297 6987 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704375975685-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704375975685-392-0/minicluster-data/master-1/wal
uuid: "952a9b1a252747a8acdcfa8e04debe30"
format_stamp: "Formatted at 2026-09-29 17:53:29 on dist-test-slave-53nh"
server_key: "9a5cdd0c7b066ae9a56f951ebddcd967"
server_key_iv: "19f0f12433f729d216cab254efb3b8dd"
server_key_version: "encryptionkey@0"
I20260929 17:53:29.575344 6987 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 952a9b1a252747a8acdcfa8e04debe30: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 17:53:29.578079 6987 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 952a9b1a252747a8acdcfa8e04debe30: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260929 17:53:29.578151 6987 log.cc:1196] T 00000000000000000000000000000000 P 952a9b1a252747a8acdcfa8e04debe30: Deleting WAL directory at /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704375975685-392-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260929 17:53:29.578395 6987 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 952a9b1a252747a8acdcfa8e04debe30: Deleting consensus metadata
I20260929 17:53:29.578545 6987 master_runner.cc:315] Copying system tablet from 127.0.98.62:39863
I20260929 17:53:29.579668 6987 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 952a9b1a252747a8acdcfa8e04debe30: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:39863
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704375975685-392-0/krb5kdc/oryx_127.0.98.60.keytab.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704375975685-392-0/krb5kdc/oryx_127.0.98.60.keytab.
I20260929 17:53:29.583693 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskup3Soe/build/debug/bin/kudu
/tmp/dist-test-taskup3Soe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704375975685-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704375975685-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704375975685-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704375975685-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:39371
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704375975685-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39863,127.0.98.61:34183,127.0.98.60:39371
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704375975685-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704375975685-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704375975685-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 17:53:29.588380 6562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:29.580279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40058 (local address 127.0.98.62:39863)
0929 17:53:29.580412 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:29.580417 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:29.580524 (+ 107us) server_negotiation.cc:407] Connection header received
0929 17:53:29.580663 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:29.580666 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:29.580713 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:29.580776 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:53:29.581264 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:29.581839 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:29.582607 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:29.582837 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:29.582945 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:29.584903 (+ 1958us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:53:29.584908 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:53:29.584911 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:53:29.584936 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:53:29.586607 (+ 1671us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:53:29.587085 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:53:29.587088 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:53:29.587090 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:53:29.587168 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:53:29.587455 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:53:29.587458 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:53:29.587460 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:53:29.587677 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:53:29.587745 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0929 17:53:29.588076 (+ 331us) server_negotiation.cc:299] Negotiation successful
0929 17:53:29.588201 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260929 17:53:29.588941 7094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:29.580223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39863 (local address 127.0.0.1:40058)
0929 17:53:29.580440 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:29.580454 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 17:53:29.580570 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:53:29.580803 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:53:29.580805 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:53:29.580990 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:53:29.581157 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:29.581163 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:29.581928 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:29.581933 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:29.582504 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:29.582512 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:29.583092 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:29.583097 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:29.583205 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:29.584267 (+ 1062us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:53:29.584291 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:53:29.584734 (+ 443us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:53:29.586742 (+ 2008us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:53:29.586745 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:53:29.586747 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:53:29.586984 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:53:29.587265 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:53:29.587268 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:53:29.587270 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:53:29.587364 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:53:29.587810 (+ 446us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:53:29.587813 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:53:29.587909 (+ 96us) client_negotiation.cc:769] Sending connection context
0929 17:53:29.588598 (+ 689us) client_negotiation.cc:240] Negotiation successful
0929 17:53:29.588751 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":83,"threads_started":1}
I20260929 17:53:29.589529 6543 tablet_copy_service.cc:140] P 14cabac4c134442b901728b1c630deb5: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 952a9b1a252747a8acdcfa8e04debe30 ({username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:40058)
I20260929 17:53:29.589751 6543 tablet_copy_service.cc:161] P 14cabac4c134442b901728b1c630deb5: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 952a9b1a252747a8acdcfa8e04debe30 at {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:40058: session id = 952a9b1a252747a8acdcfa8e04debe30-00000000000000000000000000000000
I20260929 17:53:29.591593 6543 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 14cabac4c134442b901728b1c630deb5: Tablet Copy: opened 0 blocks and 1 log segments
I20260929 17:53:29.593465 6987 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260929 17:53:29.595724 6987 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 952a9b1a252747a8acdcfa8e04debe30: tablet copy: Starting download of 0 data blocks...
I20260929 17:53:29.595908 6987 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 952a9b1a252747a8acdcfa8e04debe30: tablet copy: Starting download of 1 WAL segments...
I20260929 17:53:29.597766 6987 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 952a9b1a252747a8acdcfa8e04debe30: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260929 17:53:29.599622 6543 tablet_copy_service.cc:342] P 14cabac4c134442b901728b1c630deb5: Request end of tablet copy session 952a9b1a252747a8acdcfa8e04debe30-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:40058
I20260929 17:53:29.599757 6543 tablet_copy_service.cc:433] P 14cabac4c134442b901728b1c630deb5: ending tablet copy session 952a9b1a252747a8acdcfa8e04debe30-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 952a9b1a252747a8acdcfa8e04debe30
I20260929 17:53:29.607287 6987 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:53:29.608899 7099 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:53:29.608870 7096 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:53:29.609092 7097 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:53:29.609354 6987 server_base.cc:1034] running on GCE node
I20260929 17:53:29.609486 6987 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:53:29.609771 6987 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:53:29.610908 6987 hybrid_clock.cc:648] HybridClock initialized: now 1790704409610896 us; error 26 us; skew 500 ppm
Sep 29 17:53:29 dist-test-slave-53nh krb5kdc[6478](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704409, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:53:29.612495 6987 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260929 17:53:29.612957 6987 webserver.cc:460] Webserver started at http://127.0.98.61:36607/ using document root <none> and password file <none>
I20260929 17:53:29.613154 6987 fs_manager.cc:359] Metadata directory not provided
I20260929 17:53:29.613196 6987 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:53:29.614457 6987 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260929 17:53:29.615391 7106 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:53:29.615649 6987 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260929 17:53:29.615729 6987 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704375975685-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704375975685-392-0/minicluster-data/master-1/wal
uuid: "952a9b1a252747a8acdcfa8e04debe30"
format_stamp: "Formatted at 2026-09-29 17:53:29 on dist-test-slave-53nh"
server_key: "9a5cdd0c7b066ae9a56f951ebddcd967"
server_key_iv: "19f0f12433f729d216cab254efb3b8dd"
server_key_version: "encryptionkey@0"
I20260929 17:53:29.615806 6987 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704375975685-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704375975685-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704375975685-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:53:29.634932 6987 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:53:29.637233 6987 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:53:29.641812 6987 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:34183
I20260929 17:53:29.641919 7158 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:34183 every 8 connection(s)
I20260929 17:53:29.642374 6987 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704375975685-392-0/minicluster-data/master-1/data/info.pb
I20260929 17:53:29.644160 7159 sys_catalog.cc:263] Verifying existing consensus state
I20260929 17:53:29.645219 7159 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 952a9b1a252747a8acdcfa8e04debe30: Bootstrap starting.
I20260929 17:53:29.655616 7159 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 952a9b1a252747a8acdcfa8e04debe30: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 17:53:29.655953 7159 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 952a9b1a252747a8acdcfa8e04debe30: Bootstrap complete.
I20260929 17:53:29.656558 7159 raft_consensus.cc:348] T 00000000000000000000000000000000 P 952a9b1a252747a8acdcfa8e04debe30 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "14cabac4c134442b901728b1c630deb5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39863 } } peers { permanent_uuid: "952a9b1a252747a8acdcfa8e04debe30" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 34183 } attrs { promote: true } }
I20260929 17:53:29.656710 7159 raft_consensus.cc:729] T 00000000000000000000000000000000 P 952a9b1a252747a8acdcfa8e04debe30 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 952a9b1a252747a8acdcfa8e04debe30, State: Initialized, Role: LEARNER
I20260929 17:53:29.656828 7159 consensus_queue.cc:260] T 00000000000000000000000000000000 P 952a9b1a252747a8acdcfa8e04debe30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "14cabac4c134442b901728b1c630deb5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39863 } } peers { permanent_uuid: "952a9b1a252747a8acdcfa8e04debe30" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 34183 } attrs { promote: true } }
I20260929 17:53:29.657191 7162 sys_catalog.cc:455] T 00000000000000000000000000000000 P 952a9b1a252747a8acdcfa8e04debe30 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "14cabac4c134442b901728b1c630deb5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39863 } } peers { permanent_uuid: "952a9b1a252747a8acdcfa8e04debe30" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 34183 } attrs { promote: true } } }
I20260929 17:53:29.657321 7159 sys_catalog.cc:564] T 00000000000000000000000000000000 P 952a9b1a252747a8acdcfa8e04debe30 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 17:53:29.657367 7162 sys_catalog.cc:458] T 00000000000000000000000000000000 P 952a9b1a252747a8acdcfa8e04debe30 [sys.catalog]: This master's current role is: LEARNER
I20260929 17:53:29.660175 7173 catalog_manager.cc:1259] Loaded cluster ID: 206c4021e40644f6bf3327638ec8f5f8
I20260929 17:53:29.660293 7173 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 952a9b1a252747a8acdcfa8e04debe30: loading cluster ID for follower catalog manager: success
I20260929 17:53:29.662367 7173 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 952a9b1a252747a8acdcfa8e04debe30: acquiring CA information for follower catalog manager: success
I20260929 17:53:29.663638 7173 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 952a9b1a252747a8acdcfa8e04debe30: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260929 17:53:29.709401 7095 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:53:29.709667 7095 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:53:29.709707 7095 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:53:29.713614 7095 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 17:53:29.713668 7095 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:53:29.713686 7095 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:53:29.713701 7095 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 17:53:29.713721 7095 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 17:53:29.718978 7095 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704375975685-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704375975685-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39863,127.0.98.61:34183,127.0.98.60:39371
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704375975685-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:39371
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704375975685-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7095
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704375975685-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:53:29.719985 7095 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:53:29.720978 7095 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:53:29.729300 7179 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:53:29.729300 7178 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:53:29.729527 7095 server_base.cc:1034] running on GCE node
W20260929 17:53:29.729310 7181 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:53:29.730204 7095 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:53:29.731405 7095 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:53:29.732587 7095 hybrid_clock.cc:648] HybridClock initialized: now 1790704409732564 us; error 37 us; skew 500 ppm
Sep 29 17:53:29 dist-test-slave-53nh krb5kdc[6478](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704409, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:53:29.735471 7095 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260929 17:53:29.736424 7095 webserver.cc:460] Webserver started at http://127.0.98.60:42225/ using document root <none> and password file <none>
I20260929 17:53:29.736953 7095 fs_manager.cc:359] Metadata directory not provided
I20260929 17:53:29.737042 7095 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:53:29.737237 7095 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:53:29.739244 7095 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704375975685-392-0/minicluster-data/master-2/data/instance:
uuid: "60abdd24a53146259072b5a6b7220ec0"
format_stamp: "Formatted at 2026-09-29 17:53:29 on dist-test-slave-53nh"
server_key: "0e8113cd5e2b47637905da57dcf3c986"
server_key_iv: "9cbf1620d9ea66870f7bf1d258ef3081"
server_key_version: "encryptionkey@0"
I20260929 17:53:29.739751 7095 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704375975685-392-0/minicluster-data/master-2/wal/instance:
uuid: "60abdd24a53146259072b5a6b7220ec0"
format_stamp: "Formatted at 2026-09-29 17:53:29 on dist-test-slave-53nh"
server_key: "0e8113cd5e2b47637905da57dcf3c986"
server_key_iv: "9cbf1620d9ea66870f7bf1d258ef3081"
server_key_version: "encryptionkey@0"
I20260929 17:53:29.743434 7095 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 17:53:29.745905 7188 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:53:29.747201 7095 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 17:53:29.747329 7095 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704375975685-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704375975685-392-0/minicluster-data/master-2/wal
uuid: "60abdd24a53146259072b5a6b7220ec0"
format_stamp: "Formatted at 2026-09-29 17:53:29 on dist-test-slave-53nh"
server_key: "0e8113cd5e2b47637905da57dcf3c986"
server_key_iv: "9cbf1620d9ea66870f7bf1d258ef3081"
server_key_version: "encryptionkey@0"
I20260929 17:53:29.747432 7095 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704375975685-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704375975685-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704375975685-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:53:29.760119 7095 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:53:29.763067 7095 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:53:29.763243 7095 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:53:29.769984 7095 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:39371
I20260929 17:53:29.770222 7240 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:39371 every 8 connection(s)
I20260929 17:53:29.770922 7095 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704375975685-392-0/minicluster-data/master-2/data/info.pb
I20260929 17:53:29.773581 7241 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:53:29.777227 7241 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39863 } has no permanent_uuid. Determining permanent_uuid...
I20260929 17:53:29.779623 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskup3Soe/build/debug/bin/kudu as pid 7095
Sep 29 17:53:29 dist-test-slave-53nh krb5kdc[6478](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704409, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 29 17:53:29 dist-test-slave-53nh krb5kdc[6478](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704408, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260929 17:53:29.791313 6562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:29.779098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40066 (local address 127.0.98.62:39863)
0929 17:53:29.779240 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:29.779244 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:29.780065 (+ 821us) server_negotiation.cc:407] Connection header received
0929 17:53:29.781198 (+ 1133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:29.781202 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:29.781249 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:29.781342 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:53:29.782310 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:29.783121 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:29.784465 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:29.784831 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:29.784926 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:29.787719 (+ 2793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:53:29.787723 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:53:29.787726 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:53:29.787754 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:53:29.789283 (+ 1529us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:53:29.789837 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:53:29.789841 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:53:29.789843 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:53:29.789925 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:53:29.790255 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:53:29.790258 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:53:29.790259 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:53:29.790463 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:53:29.790543 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0929 17:53:29.791051 (+ 508us) server_negotiation.cc:299] Negotiation successful
0929 17:53:29.791171 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 17:53:29.792650 7244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:29.780314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46784 (local address 127.0.98.60:39371)
0929 17:53:29.780618 (+ 304us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:29.780628 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:29.780748 (+ 120us) server_negotiation.cc:407] Connection header received
0929 17:53:29.780982 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:29.781013 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:29.781273 (+ 260us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:29.781442 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:53:29.782225 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:29.783414 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:29.784495 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:29.785062 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:29.785143 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:29.788127 (+ 2984us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:53:29.788134 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:53:29.788148 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:53:29.788171 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:53:29.789773 (+ 1602us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:53:29.790209 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:53:29.790214 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:53:29.790215 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:53:29.790297 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:53:29.790537 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:53:29.790540 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:53:29.790542 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:53:29.790801 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:53:29.790942 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0929 17:53:29.791503 (+ 561us) server_negotiation.cc:299] Negotiation successful
0929 17:53:29.791630 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":213,"thread_start_us":75,"threads_started":1}
I20260929 17:53:29.792651 7243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:29.779431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39863 (local address 127.0.0.1:40066)
0929 17:53:29.779905 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:29.779958 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0929 17:53:29.780988 (+ 1030us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:53:29.781393 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:53:29.781399 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:53:29.781829 (+ 430us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:53:29.782172 (+ 343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:29.782191 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:29.783228 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:29.783230 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:29.784352 (+ 1122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:29.784362 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:29.784970 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:29.784974 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:29.785141 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:29.785847 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:53:29.785865 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:53:29.787549 (+ 1684us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:53:29.789381 (+ 1832us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:53:29.789388 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:53:29.789396 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:53:29.789733 (+ 337us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:53:29.789998 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:53:29.790001 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:53:29.790003 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:53:29.790176 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:53:29.790556 (+ 380us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:53:29.790562 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:53:29.790826 (+ 264us) client_negotiation.cc:769] Sending connection context
0929 17:53:29.790997 (+ 171us) client_negotiation.cc:240] Negotiation successful
0929 17:53:29.791601 (+ 604us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":105,"threads_started":1}
I20260929 17:53:29.794450 7241 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34183 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 17:53:29 dist-test-slave-53nh krb5kdc[6478](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704409, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260929 17:53:29.801982 7243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:29.794902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34183 (local address 127.0.0.1:41462)
0929 17:53:29.795011 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:29.795022 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 17:53:29.795081 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:53:29.795421 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:53:29.795423 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:53:29.795548 (+ 125us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:53:29.795636 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:29.795641 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:29.796537 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:29.796539 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:29.797112 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:29.797119 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:29.797533 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:29.797537 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:29.797636 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:29.798056 (+ 420us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:53:29.798066 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:53:29.799177 (+ 1111us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:53:29.800794 (+ 1617us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:53:29.800797 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:53:29.800798 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:53:29.800969 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:53:29.801233 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:53:29.801236 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:53:29.801237 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:53:29.801314 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:53:29.801602 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:53:29.801604 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:53:29.801679 (+ 75us) client_negotiation.cc:769] Sending connection context
0929 17:53:29.801749 (+ 70us) client_negotiation.cc:240] Negotiation successful
0929 17:53:29.801860 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260929 17:53:29.802057 7245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:29.795022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41462 (local address 127.0.98.61:34183)
0929 17:53:29.795249 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:29.795252 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:29.795264 (+ 12us) server_negotiation.cc:407] Connection header received
0929 17:53:29.795297 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:29.795300 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:29.795336 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:29.795405 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:53:29.795758 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:29.796454 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:29.797215 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:29.797444 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:29.797518 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:29.799302 (+ 1784us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:53:29.799304 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:53:29.799307 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:53:29.799330 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:53:29.800709 (+ 1379us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:53:29.801079 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:53:29.801082 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:53:29.801083 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:53:29.801152 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:53:29.801377 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:53:29.801379 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:53:29.801380 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:53:29.801542 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:53:29.801608 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0929 17:53:29.801789 (+ 181us) server_negotiation.cc:299] Negotiation successful
0929 17:53:29.801936 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":84,"threads_started":1}
I20260929 17:53:29.803373 7241 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39371 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 17:53:29 dist-test-slave-53nh krb5kdc[6478](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704409, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260929 17:53:29.810135 7243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:29.803844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39371 (local address 127.0.0.1:46794)
0929 17:53:29.803932 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:29.803946 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 17:53:29.804029 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:53:29.804248 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:53:29.804250 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:53:29.804376 (+ 126us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:53:29.804446 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:29.804451 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:29.805101 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:29.805103 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:29.805595 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:29.805602 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:29.805999 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:29.806001 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:29.806055 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:29.806471 (+ 416us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:53:29.806483 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:53:29.807681 (+ 1198us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:53:29.809069 (+ 1388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:53:29.809072 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:53:29.809074 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:53:29.809253 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:53:29.809455 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:53:29.809457 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:53:29.809458 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:53:29.809520 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:53:29.809776 (+ 256us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:53:29.809778 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:53:29.809843 (+ 65us) client_negotiation.cc:769] Sending connection context
0929 17:53:29.809913 (+ 70us) client_negotiation.cc:240] Negotiation successful
0929 17:53:29.809998 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260929 17:53:29.810161 7244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:29.803895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46794 (local address 127.0.98.60:39371)
0929 17:53:29.803996 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:29.803999 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:29.804010 (+ 11us) server_negotiation.cc:407] Connection header received
0929 17:53:29.804121 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:29.804123 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:29.804157 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:29.804212 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:53:29.804522 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:29.805022 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:29.805702 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:29.805928 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:29.805993 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:29.807796 (+ 1803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:53:29.807799 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:53:29.807801 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:53:29.807821 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:53:29.808949 (+ 1128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:53:29.809329 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:53:29.809332 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:53:29.809333 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:53:29.809395 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:53:29.809574 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:53:29.809576 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:53:29.809577 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:53:29.809716 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:53:29.809780 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0929 17:53:29.809962 (+ 182us) server_negotiation.cc:299] Negotiation successful
0929 17:53:29.810043 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260929 17:53:29.813292 7241 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 60abdd24a53146259072b5a6b7220ec0: Bootstrap starting.
I20260929 17:53:29.815438 7241 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 60abdd24a53146259072b5a6b7220ec0: Neither blocks nor log segments found. Creating new log.
I20260929 17:53:29.816145 7241 log.cc:824] T 00000000000000000000000000000000 P 60abdd24a53146259072b5a6b7220ec0: Log is configured to *not* fsync() on all Append() calls
I20260929 17:53:29.817917 7241 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 60abdd24a53146259072b5a6b7220ec0: No bootstrap required, opened a new log
I20260929 17:53:29.820822 7241 raft_consensus.cc:348] T 00000000000000000000000000000000 P 60abdd24a53146259072b5a6b7220ec0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14cabac4c134442b901728b1c630deb5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39863 } } peers { permanent_uuid: "952a9b1a252747a8acdcfa8e04debe30" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34183 } } peers { permanent_uuid: "60abdd24a53146259072b5a6b7220ec0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39371 } }
I20260929 17:53:29.821044 7241 raft_consensus.cc:374] T 00000000000000000000000000000000 P 60abdd24a53146259072b5a6b7220ec0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:53:29.821094 7241 raft_consensus.cc:729] T 00000000000000000000000000000000 P 60abdd24a53146259072b5a6b7220ec0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60abdd24a53146259072b5a6b7220ec0, State: Initialized, Role: FOLLOWER
I20260929 17:53:29.821455 7241 consensus_queue.cc:260] T 00000000000000000000000000000000 P 60abdd24a53146259072b5a6b7220ec0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14cabac4c134442b901728b1c630deb5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39863 } } peers { permanent_uuid: "952a9b1a252747a8acdcfa8e04debe30" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34183 } } peers { permanent_uuid: "60abdd24a53146259072b5a6b7220ec0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39371 } }
I20260929 17:53:29.821820 7247 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60abdd24a53146259072b5a6b7220ec0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14cabac4c134442b901728b1c630deb5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39863 } } peers { permanent_uuid: "952a9b1a252747a8acdcfa8e04debe30" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34183 } } peers { permanent_uuid: "60abdd24a53146259072b5a6b7220ec0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39371 } } }
I20260929 17:53:29.821930 7247 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60abdd24a53146259072b5a6b7220ec0 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 17:53:29.822503 7241 sys_catalog.cc:564] T 00000000000000000000000000000000 P 60abdd24a53146259072b5a6b7220ec0 [sys.catalog]: configured and running, proceeding with master startup.
W20260929 17:53:29.826759 7258 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 60abdd24a53146259072b5a6b7220ec0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 17:53:29.826833 7258 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 17:53:29.829731 7095 master_runner.cc:428] Detected that this master 60abdd24a53146259072b5a6b7220ec0 is joining an existing cluster
I20260929 17:53:29.829777 7095 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.60:39371
I20260929 17:53:29.839653 7245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:29.831972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41476 (local address 127.0.98.61:34183)
0929 17:53:29.832078 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:29.832082 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:29.832614 (+ 532us) server_negotiation.cc:407] Connection header received
0929 17:53:29.832763 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:29.832766 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:29.832809 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:29.832847 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:53:29.833580 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:29.834097 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:29.835163 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:29.835491 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:29.835563 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:29.836780 (+ 1217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:53:29.836783 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:53:29.836786 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:53:29.836814 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:53:29.838173 (+ 1359us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:53:29.838578 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:53:29.838581 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:53:29.838584 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:53:29.838655 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:53:29.838889 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:53:29.838893 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:53:29.838894 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:53:29.839072 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:53:29.839151 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0929 17:53:29.839409 (+ 258us) server_negotiation.cc:299] Negotiation successful
0929 17:53:29.839515 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260929 17:53:29.839669 7264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:29.831636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34183 (local address 127.0.0.1:41476)
0929 17:53:29.832539 (+ 903us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:29.832553 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 17:53:29.832668 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:53:29.832972 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:53:29.832974 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:53:29.833230 (+ 256us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:53:29.833454 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:29.833460 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:29.834201 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:29.834204 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:29.835042 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:29.835051 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:29.835706 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:29.835709 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:29.835790 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:29.836326 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:53:29.836339 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:53:29.836659 (+ 320us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:53:29.838289 (+ 1630us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:53:29.838292 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:53:29.838294 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:53:29.838494 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:53:29.838735 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:53:29.838738 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:53:29.838740 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:53:29.838816 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:53:29.839153 (+ 337us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:53:29.839156 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:53:29.839256 (+ 100us) client_negotiation.cc:769] Sending connection context
0929 17:53:29.839349 (+ 93us) client_negotiation.cc:240] Negotiation successful
0929 17:53:29.839483 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":822,"mutex_wait_us":42,"spinlock_wait_cycles":24320,"thread_start_us":89,"threads_started":1}
I20260929 17:53:29.845901 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskup3Soe/build/debug/bin/kudu with pid 6585
I20260929 17:53:29.846683 6562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:29.832300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40078 (local address 127.0.98.62:39863)
0929 17:53:29.832411 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:29.832415 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:29.840085 (+ 7670us) server_negotiation.cc:407] Connection header received
0929 17:53:29.840176 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:29.840180 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:29.840245 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:29.840330 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:53:29.840729 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:29.841365 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:29.842161 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:29.842382 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:29.842466 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:29.843473 (+ 1007us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:53:29.843477 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:53:29.843480 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:53:29.843505 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:53:29.844706 (+ 1201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:53:29.845323 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:53:29.845327 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:53:29.845329 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:53:29.845427 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:53:29.845744 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:53:29.845748 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:53:29.845750 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:53:29.845975 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:53:29.846045 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0929 17:53:29.846391 (+ 346us) server_negotiation.cc:299] Negotiation successful
0929 17:53:29.846520 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 17:53:29.847141 7265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:29.832237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39863 (local address 127.0.0.1:40078)
0929 17:53:29.839985 (+ 7748us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:29.839998 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 17:53:29.840094 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:53:29.840354 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:53:29.840356 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:53:29.840535 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:53:29.840621 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:29.840626 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:29.841463 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:29.841466 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:29.842057 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:29.842065 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:29.842487 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:29.842490 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:29.842580 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:29.843033 (+ 453us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:53:29.843044 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:53:29.843339 (+ 295us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:53:29.844804 (+ 1465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:53:29.844807 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:53:29.844809 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:53:29.845046 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:53:29.845534 (+ 488us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:53:29.845537 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:53:29.845539 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:53:29.845637 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:53:29.846114 (+ 477us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:53:29.846117 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:53:29.846205 (+ 88us) client_negotiation.cc:769] Sending connection context
0929 17:53:29.846826 (+ 621us) client_negotiation.cc:240] Negotiation successful
0929 17:53:29.846974 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7680,"thread_start_us":88,"threads_started":1}
I20260929 17:53:29.853176 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskup3Soe/build/debug/bin/kudu
/tmp/dist-test-taskup3Soe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704375975685-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704375975685-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704375975685-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704375975685-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:46193
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:39863
--webserver_port=41565
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704375975685-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:39863,127.0.98.61:34183
--tserver_master_addrs=127.0.98.62:39863,127.0.98.61:34183,127.0.98.60:39371 with env {KRB5CCNAME=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704375975685-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704375975685-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704375975685-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 17:53:29.857153 7266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:29.831906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39371 (local address 127.0.0.1:46798)
0929 17:53:29.832551 (+ 645us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:29.832561 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 17:53:29.832654 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:53:29.832906 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:53:29.832909 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:53:29.833220 (+ 311us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:53:29.833448 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:29.833456 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:29.834484 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:29.834487 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:29.835288 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:29.835295 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:29.835840 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:29.835843 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:29.835921 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:29.853679 (+ 17758us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:53:29.853692 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:53:29.854013 (+ 321us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:53:29.855663 (+ 1650us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:53:29.855666 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:53:29.855668 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:53:29.855843 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:53:29.856066 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:53:29.856069 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:53:29.856071 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:53:29.856149 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:53:29.856603 (+ 454us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:53:29.856606 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:53:29.856678 (+ 72us) client_negotiation.cc:769] Sending connection context
0929 17:53:29.856771 (+ 93us) client_negotiation.cc:240] Negotiation successful
0929 17:53:29.856911 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":599,"mutex_wait_us":40,"spinlock_wait_cycles":5504,"thread_start_us":82,"threads_started":1}
I20260929 17:53:29.857187 7244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:29.831736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46798 (local address 127.0.98.60:39371)
0929 17:53:29.831831 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:29.831835 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:29.832637 (+ 802us) server_negotiation.cc:407] Connection header received
0929 17:53:29.832750 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:29.832753 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:29.832800 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:29.832884 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:53:29.833562 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:29.834341 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:29.835385 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:29.835749 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:29.835848 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:29.854136 (+ 18288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:53:29.854140 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:53:29.854142 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:53:29.854162 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:53:29.855566 (+ 1404us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:53:29.855922 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:53:29.855925 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:53:29.855927 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:53:29.855995 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:53:29.856252 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:53:29.856255 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:53:29.856257 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:53:29.856484 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:53:29.856571 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0929 17:53:29.856841 (+ 270us) server_negotiation.cc:299] Negotiation successful
0929 17:53:29.856983 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":23,"server-negotiator.queue_time_us":35}
Sep 29 17:53:29 dist-test-slave-53nh krb5kdc[6478](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704408, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260929 17:53:29.862105 7080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:29.851481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39371 (local address 127.0.0.1:46800)
0929 17:53:29.851609 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:29.851623 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 17:53:29.851710 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:53:29.853206 (+ 1496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:53:29.853209 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:53:29.853417 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:53:29.853536 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:29.853543 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:29.855133 (+ 1590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:29.855136 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:29.856010 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:29.856021 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:29.856537 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:29.856540 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:29.856635 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:29.857064 (+ 429us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:53:29.857077 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:53:29.858598 (+ 1521us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:53:29.860634 (+ 2036us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:53:29.860638 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:53:29.860640 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:53:29.860876 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:53:29.861227 (+ 351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:53:29.861230 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:53:29.861232 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:53:29.861312 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:53:29.861638 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:53:29.861641 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:53:29.861728 (+ 87us) client_negotiation.cc:769] Sending connection context
0929 17:53:29.861809 (+ 81us) client_negotiation.cc:240] Negotiation successful
0929 17:53:29.861952 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260929 17:53:29.862120 7267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:29.851704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46800 (local address 127.0.98.60:39371)
0929 17:53:29.852829 (+ 1125us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:29.852835 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:29.852846 (+ 11us) server_negotiation.cc:407] Connection header received
0929 17:53:29.852904 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:29.852906 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:29.852949 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:29.854126 (+ 1177us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:53:29.854240 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:29.855004 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:29.856117 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:29.856425 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:29.856524 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:29.858716 (+ 2192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:53:29.858718 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:53:29.858720 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:53:29.858738 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:53:29.860538 (+ 1800us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:53:29.860957 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:53:29.860960 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:53:29.860961 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:53:29.861134 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:53:29.861388 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:53:29.861391 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:53:29.861393 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:53:29.861556 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:53:29.861634 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0929 17:53:29.861862 (+ 228us) server_negotiation.cc:299] Negotiation successful
0929 17:53:29.861956 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1068,"thread_start_us":105,"threads_started":1}
I20260929 17:53:29.863294 6523 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.60:39371: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "60abdd24a53146259072b5a6b7220ec0"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.60"
port: 39371
}
attrs {
promote: true
}
}
dest_uuid: "14cabac4c134442b901728b1c630deb5"
cas_config_opid_index: 5
I20260929 17:53:29.863936 6523 consensus_queue.cc:237] T 00000000000000000000000000000000 P 14cabac4c134442b901728b1c630deb5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "14cabac4c134442b901728b1c630deb5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39863 } } peers { permanent_uuid: "952a9b1a252747a8acdcfa8e04debe30" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 34183 } attrs { promote: true } } peers { permanent_uuid: "60abdd24a53146259072b5a6b7220ec0" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39371 } attrs { promote: true } }
I20260929 17:53:29.864974 7082 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 14cabac4c134442b901728b1c630deb5 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 60abdd24a53146259072b5a6b7220ec0 (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "14cabac4c134442b901728b1c630deb5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39863 } } peers { permanent_uuid: "952a9b1a252747a8acdcfa8e04debe30" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 34183 } attrs { promote: true } } peers { permanent_uuid: "60abdd24a53146259072b5a6b7220ec0" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39371 } attrs { promote: true } } }
I20260929 17:53:29.866444 7082 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.60:39371
I20260929 17:53:29.866603 7082 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14cabac4c134442b901728b1c630deb5 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "14cabac4c134442b901728b1c630deb5" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "14cabac4c134442b901728b1c630deb5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39863 } } peers { permanent_uuid: "952a9b1a252747a8acdcfa8e04debe30" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 34183 } attrs { promote: true } } peers { permanent_uuid: "60abdd24a53146259072b5a6b7220ec0" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39371 } attrs { promote: true } } }
I20260929 17:53:29.866688 7082 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14cabac4c134442b901728b1c630deb5 [sys.catalog]: This master's current role is: LEADER
I20260929 17:53:29.867080 7095 master.cc:537] Master@127.0.98.60:39371 shutting down...
I20260929 17:53:29.869936 7080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:29.865171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34183 (local address 127.0.0.1:41488)
0929 17:53:29.865273 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:29.865286 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 17:53:29.865354 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:53:29.865962 (+ 608us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:53:29.865965 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:53:29.865983 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:53:29.866067 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:29.866073 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:29.867095 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:29.867099 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:29.868325 (+ 1226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:29.868336 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:29.869496 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:29.869500 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:29.869594 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:29.869643 (+ 49us) client_negotiation.cc:769] Sending connection context
0929 17:53:29.869730 (+ 87us) client_negotiation.cc:240] Negotiation successful
0929 17:53:29.869805 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260929 17:53:29.870281 7245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:29.865479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41488 (local address 127.0.98.61:34183)
0929 17:53:29.865595 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:29.865599 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:29.865613 (+ 14us) server_negotiation.cc:407] Connection header received
0929 17:53:29.865661 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:29.865665 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:29.865742 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:29.865837 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:53:29.866191 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:29.866966 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:29.868470 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:29.869163 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:29.869262 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:29.869432 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0929 17:53:29.870083 (+ 651us) server_negotiation.cc:299] Negotiation successful
0929 17:53:29.870164 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
W20260929 17:53:29.870560 6508 consensus_peers.cc:487] T 00000000000000000000000000000000 P 14cabac4c134442b901728b1c630deb5 -> Peer 60abdd24a53146259072b5a6b7220ec0 (127.0.98.60:39371): Couldn't send request to peer 60abdd24a53146259072b5a6b7220ec0. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260929 17:53:29.870711 7133 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 952a9b1a252747a8acdcfa8e04debe30 [term 1 LEARNER]: Refusing update from remote peer 14cabac4c134442b901728b1c630deb5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260929 17:53:29.871178 7083 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 14cabac4c134442b901728b1c630deb5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "952a9b1a252747a8acdcfa8e04debe30" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 34183 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260929 17:53:29.872118 7095 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 60abdd24a53146259072b5a6b7220ec0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260929 17:53:29.872406 7095 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 60abdd24a53146259072b5a6b7220ec0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260929 17:53:29.872498 7095 tablet_replica.cc:330] T 00000000000000000000000000000000 P 60abdd24a53146259072b5a6b7220ec0: stopping tablet replica
I20260929 17:53:29.873124 7133 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 952a9b1a252747a8acdcfa8e04debe30 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 60abdd24a53146259072b5a6b7220ec0 (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "14cabac4c134442b901728b1c630deb5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39863 } } peers { permanent_uuid: "952a9b1a252747a8acdcfa8e04debe30" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 34183 } attrs { promote: true } } peers { permanent_uuid: "60abdd24a53146259072b5a6b7220ec0" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39371 } attrs { promote: true } } }
I20260929 17:53:29.874948 7270 sys_catalog.cc:455] T 00000000000000000000000000000000 P 952a9b1a252747a8acdcfa8e04debe30 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "14cabac4c134442b901728b1c630deb5" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "14cabac4c134442b901728b1c630deb5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39863 } } peers { permanent_uuid: "952a9b1a252747a8acdcfa8e04debe30" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 34183 } attrs { promote: true } } peers { permanent_uuid: "60abdd24a53146259072b5a6b7220ec0" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39371 } attrs { promote: true } } }
I20260929 17:53:29.875115 7270 sys_catalog.cc:458] T 00000000000000000000000000000000 P 952a9b1a252747a8acdcfa8e04debe30 [sys.catalog]: This master's current role is: LEARNER
I20260929 17:53:29.875384 7162 sys_catalog.cc:455] T 00000000000000000000000000000000 P 952a9b1a252747a8acdcfa8e04debe30 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 14cabac4c134442b901728b1c630deb5. Latest consensus state: current_term: 1 leader_uuid: "14cabac4c134442b901728b1c630deb5" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "14cabac4c134442b901728b1c630deb5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39863 } } peers { permanent_uuid: "952a9b1a252747a8acdcfa8e04debe30" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 34183 } attrs { promote: true } } peers { permanent_uuid: "60abdd24a53146259072b5a6b7220ec0" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39371 } attrs { promote: true } } }
I20260929 17:53:29.875504 7162 sys_catalog.cc:458] T 00000000000000000000000000000000 P 952a9b1a252747a8acdcfa8e04debe30 [sys.catalog]: This master's current role is: LEARNER
I20260929 17:53:29.876197 7083 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 14cabac4c134442b901728b1c630deb5: attempting to promote NON_VOTER 952a9b1a252747a8acdcfa8e04debe30 to VOTER
I20260929 17:53:29.876621 7083 consensus_queue.cc:237] T 00000000000000000000000000000000 P 14cabac4c134442b901728b1c630deb5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "14cabac4c134442b901728b1c630deb5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39863 } } peers { permanent_uuid: "952a9b1a252747a8acdcfa8e04debe30" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34183 } attrs { promote: false } } peers { permanent_uuid: "60abdd24a53146259072b5a6b7220ec0" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39371 } attrs { promote: true } }
I20260929 17:53:29.877980 7133 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 952a9b1a252747a8acdcfa8e04debe30 [term 1 LEARNER]: Refusing update from remote peer 14cabac4c134442b901728b1c630deb5: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260929 17:53:29.878325 7082 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 14cabac4c134442b901728b1c630deb5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "952a9b1a252747a8acdcfa8e04debe30" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34183 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260929 17:53:29.878940 7080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:29.878655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39371 (local address 127.0.0.1:46816)
0929 17:53:29.878814 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:29.878873 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
W20260929 17:53:29.879424 6508 consensus_peers.cc:487] T 00000000000000000000000000000000 P 14cabac4c134442b901728b1c630deb5 -> Peer 60abdd24a53146259072b5a6b7220ec0 (127.0.98.60:39371): Couldn't send request to peer 60abdd24a53146259072b5a6b7220ec0. Status: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39371: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260929 17:53:29.880388 7083 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 14cabac4c134442b901728b1c630deb5 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 952a9b1a252747a8acdcfa8e04debe30 (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "14cabac4c134442b901728b1c630deb5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39863 } } peers { permanent_uuid: "952a9b1a252747a8acdcfa8e04debe30" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34183 } attrs { promote: false } } peers { permanent_uuid: "60abdd24a53146259072b5a6b7220ec0" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39371 } attrs { promote: true } } }
I20260929 17:53:29.880825 7133 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 952a9b1a252747a8acdcfa8e04debe30 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 952a9b1a252747a8acdcfa8e04debe30 (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "14cabac4c134442b901728b1c630deb5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39863 } } peers { permanent_uuid: "952a9b1a252747a8acdcfa8e04debe30" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34183 } attrs { promote: false } } peers { permanent_uuid: "60abdd24a53146259072b5a6b7220ec0" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39371 } attrs { promote: true } } }
I20260929 17:53:29.881729 7083 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14cabac4c134442b901728b1c630deb5 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "14cabac4c134442b901728b1c630deb5" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "14cabac4c134442b901728b1c630deb5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39863 } } peers { permanent_uuid: "952a9b1a252747a8acdcfa8e04debe30" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34183 } attrs { promote: false } } peers { permanent_uuid: "60abdd24a53146259072b5a6b7220ec0" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39371 } attrs { promote: true } } }
I20260929 17:53:29.881824 7083 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14cabac4c134442b901728b1c630deb5 [sys.catalog]: This master's current role is: LEADER
I20260929 17:53:29.881929 7083 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14cabac4c134442b901728b1c630deb5 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "14cabac4c134442b901728b1c630deb5" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "14cabac4c134442b901728b1c630deb5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39863 } } peers { permanent_uuid: "952a9b1a252747a8acdcfa8e04debe30" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34183 } attrs { promote: false } } peers { permanent_uuid: "60abdd24a53146259072b5a6b7220ec0" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39371 } attrs { promote: true } } }
I20260929 17:53:29.881989 7083 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14cabac4c134442b901728b1c630deb5 [sys.catalog]: This master's current role is: LEADER
I20260929 17:53:29.882471 7162 sys_catalog.cc:455] T 00000000000000000000000000000000 P 952a9b1a252747a8acdcfa8e04debe30 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "14cabac4c134442b901728b1c630deb5" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "14cabac4c134442b901728b1c630deb5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39863 } } peers { permanent_uuid: "952a9b1a252747a8acdcfa8e04debe30" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34183 } attrs { promote: false } } peers { permanent_uuid: "60abdd24a53146259072b5a6b7220ec0" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39371 } attrs { promote: true } } }
I20260929 17:53:29.882568 7162 sys_catalog.cc:458] T 00000000000000000000000000000000 P 952a9b1a252747a8acdcfa8e04debe30 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 17:53:29.890938 7095 master.cc:552] Master@127.0.98.60:39371 shutdown complete.
I20260929 17:53:29.890997 7095 master_runner.cc:305] Clearing existing system tablet
I20260929 17:53:29.891219 7095 fs_manager.cc:359] Metadata directory not provided
I20260929 17:53:29.891273 7095 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:53:29.892494 7095 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 17:53:29.893406 7273 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:53:29.893641 7095 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704375975685-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:53:29.893772 7095 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 17:53:29.893812 7095 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704375975685-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704375975685-392-0/minicluster-data/master-2/wal
uuid: "60abdd24a53146259072b5a6b7220ec0"
format_stamp: "Formatted at 2026-09-29 17:53:29 on dist-test-slave-53nh"
server_key: "0e8113cd5e2b47637905da57dcf3c986"
server_key_iv: "9cbf1620d9ea66870f7bf1d258ef3081"
server_key_version: "encryptionkey@0"
I20260929 17:53:29.895763 7095 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 60abdd24a53146259072b5a6b7220ec0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 17:53:29.898064 7095 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 60abdd24a53146259072b5a6b7220ec0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260929 17:53:29.898147 7095 log.cc:1196] T 00000000000000000000000000000000 P 60abdd24a53146259072b5a6b7220ec0: Deleting WAL directory at /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704375975685-392-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260929 17:53:29.898365 7095 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 60abdd24a53146259072b5a6b7220ec0: Deleting consensus metadata
I20260929 17:53:29.898645 7095 master_runner.cc:315] Copying system tablet from 127.0.98.62:39863
I20260929 17:53:29.899719 7095 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 60abdd24a53146259072b5a6b7220ec0: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:39863
I20260929 17:53:29.908488 6562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:29.900326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40088 (local address 127.0.98.62:39863)
0929 17:53:29.900450 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:29.900455 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:29.900627 (+ 172us) server_negotiation.cc:407] Connection header received
0929 17:53:29.900809 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:29.900813 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:29.900865 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:29.900958 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:53:29.901470 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:29.902284 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:29.903423 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:29.903748 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:29.903858 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:29.905118 (+ 1260us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:53:29.905121 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:53:29.905124 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:53:29.905151 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:53:29.906727 (+ 1576us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:53:29.907169 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:53:29.907172 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:53:29.907174 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:53:29.907259 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:53:29.907515 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:53:29.907518 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:53:29.907520 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:53:29.907726 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:53:29.907879 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0929 17:53:29.908205 (+ 326us) server_negotiation.cc:299] Negotiation successful
0929 17:53:29.908327 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 17:53:29.908802 7279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:29.900225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39863 (local address 127.0.0.1:40088)
0929 17:53:29.900535 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:29.900550 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 17:53:29.900702 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:53:29.900980 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:53:29.900982 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:53:29.901216 (+ 234us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:53:29.901354 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:29.901361 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:29.902398 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:29.902402 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:29.903309 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:29.903319 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:29.903881 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:29.903885 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:29.903981 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:29.904510 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:53:29.904523 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:53:29.904950 (+ 427us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:53:29.906832 (+ 1882us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:53:29.906835 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:53:29.906837 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:53:29.907077 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:53:29.907343 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:53:29.907346 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:53:29.907348 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:53:29.907434 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:53:29.907926 (+ 492us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:53:29.907929 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:53:29.908025 (+ 96us) client_negotiation.cc:769] Sending connection context
0929 17:53:29.908127 (+ 102us) client_negotiation.cc:240] Negotiation successful
0929 17:53:29.908264 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":96,"threads_started":1}
I20260929 17:53:29.909349 6543 tablet_copy_service.cc:140] P 14cabac4c134442b901728b1c630deb5: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 60abdd24a53146259072b5a6b7220ec0 ({username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:40088)
I20260929 17:53:29.909451 6543 tablet_copy_service.cc:161] P 14cabac4c134442b901728b1c630deb5: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 60abdd24a53146259072b5a6b7220ec0 at {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:40088: session id = 60abdd24a53146259072b5a6b7220ec0-00000000000000000000000000000000
I20260929 17:53:29.910473 6543 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 14cabac4c134442b901728b1c630deb5: Tablet Copy: opened 0 blocks and 1 log segments
I20260929 17:53:29.911589 7095 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260929 17:53:29.913827 7095 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 60abdd24a53146259072b5a6b7220ec0: tablet copy: Starting download of 0 data blocks...
I20260929 17:53:29.913995 7095 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 60abdd24a53146259072b5a6b7220ec0: tablet copy: Starting download of 1 WAL segments...
I20260929 17:53:29.915446 7095 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 60abdd24a53146259072b5a6b7220ec0: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260929 17:53:29.917120 6543 tablet_copy_service.cc:342] P 14cabac4c134442b901728b1c630deb5: Request end of tablet copy session 60abdd24a53146259072b5a6b7220ec0-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:40088
I20260929 17:53:29.917212 6543 tablet_copy_service.cc:433] P 14cabac4c134442b901728b1c630deb5: ending tablet copy session 60abdd24a53146259072b5a6b7220ec0-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 60abdd24a53146259072b5a6b7220ec0
I20260929 17:53:29.924285 7095 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:53:29.925987 7283 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:53:29.926110 7280 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:53:29.926343 7095 server_base.cc:1034] running on GCE node
W20260929 17:53:29.935621 7281 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:53:29.935989 7095 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:53:29.936327 7095 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:53:29.937469 7095 hybrid_clock.cc:648] HybridClock initialized: now 1790704409937451 us; error 42 us; skew 500 ppm
Sep 29 17:53:29 dist-test-slave-53nh krb5kdc[6478](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704409, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:53:29.939119 7095 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260929 17:53:29.939611 7095 webserver.cc:460] Webserver started at http://127.0.98.60:45771/ using document root <none> and password file <none>
I20260929 17:53:29.939807 7095 fs_manager.cc:359] Metadata directory not provided
I20260929 17:53:29.939849 7095 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:53:29.941249 7095 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260929 17:53:29.942248 7290 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:53:29.942513 7095 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260929 17:53:29.942598 7095 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704375975685-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704375975685-392-0/minicluster-data/master-2/wal
uuid: "60abdd24a53146259072b5a6b7220ec0"
format_stamp: "Formatted at 2026-09-29 17:53:29 on dist-test-slave-53nh"
server_key: "0e8113cd5e2b47637905da57dcf3c986"
server_key_iv: "9cbf1620d9ea66870f7bf1d258ef3081"
server_key_version: "encryptionkey@0"
I20260929 17:53:29.942687 7095 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704375975685-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704375975685-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704375975685-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:53:29.953029 7095 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:53:29.955154 7095 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:53:29.960263 7095 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:39371
I20260929 17:53:29.960295 7342 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:39371 every 8 connection(s)
I20260929 17:53:29.960860 7095 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704375975685-392-0/minicluster-data/master-2/data/info.pb
I20260929 17:53:29.962587 7343 sys_catalog.cc:263] Verifying existing consensus state
I20260929 17:53:29.963605 7343 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 60abdd24a53146259072b5a6b7220ec0: Bootstrap starting.
I20260929 17:53:29.974849 7343 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 60abdd24a53146259072b5a6b7220ec0: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 17:53:29.975140 7343 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 60abdd24a53146259072b5a6b7220ec0: Bootstrap complete.
I20260929 17:53:29.975731 7343 raft_consensus.cc:348] T 00000000000000000000000000000000 P 60abdd24a53146259072b5a6b7220ec0 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "14cabac4c134442b901728b1c630deb5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39863 } } peers { permanent_uuid: "952a9b1a252747a8acdcfa8e04debe30" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34183 } attrs { promote: false } } peers { permanent_uuid: "60abdd24a53146259072b5a6b7220ec0" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39371 } attrs { promote: true } }
I20260929 17:53:29.975878 7343 raft_consensus.cc:729] T 00000000000000000000000000000000 P 60abdd24a53146259072b5a6b7220ec0 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 60abdd24a53146259072b5a6b7220ec0, State: Initialized, Role: LEARNER
I20260929 17:53:29.975986 7343 consensus_queue.cc:260] T 00000000000000000000000000000000 P 60abdd24a53146259072b5a6b7220ec0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "14cabac4c134442b901728b1c630deb5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39863 } } peers { permanent_uuid: "952a9b1a252747a8acdcfa8e04debe30" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34183 } attrs { promote: false } } peers { permanent_uuid: "60abdd24a53146259072b5a6b7220ec0" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39371 } attrs { promote: true } }
I20260929 17:53:29.976315 7346 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60abdd24a53146259072b5a6b7220ec0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "14cabac4c134442b901728b1c630deb5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39863 } } peers { permanent_uuid: "952a9b1a252747a8acdcfa8e04debe30" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34183 } attrs { promote: false } } peers { permanent_uuid: "60abdd24a53146259072b5a6b7220ec0" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39371 } attrs { promote: true } } }
I20260929 17:53:29.976426 7343 sys_catalog.cc:564] T 00000000000000000000000000000000 P 60abdd24a53146259072b5a6b7220ec0 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 17:53:29.976505 7346 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60abdd24a53146259072b5a6b7220ec0 [sys.catalog]: This master's current role is: LEARNER
I20260929 17:53:29.980394 7357 catalog_manager.cc:1259] Loaded cluster ID: 206c4021e40644f6bf3327638ec8f5f8
I20260929 17:53:29.980461 7357 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 60abdd24a53146259072b5a6b7220ec0: loading cluster ID for follower catalog manager: success
I20260929 17:53:29.982439 7357 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 60abdd24a53146259072b5a6b7220ec0: acquiring CA information for follower catalog manager: success
I20260929 17:53:29.983758 7357 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 60abdd24a53146259072b5a6b7220ec0: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260929 17:53:29.998121 7268 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:53:29.998454 7268 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:53:29.998543 7268 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:53:30.002935 7268 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:53:30.003026 7268 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:53:30.003129 7268 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 17:53:30.009176 7268 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704375975685-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704375975685-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704375975685-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:46193
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704375975685-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=41565
--tserver_master_addrs=127.0.98.62:39863,127.0.98.61:34183,127.0.98.60:39371
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7268
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704375975685-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:53:30.010318 7268 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:53:30.011356 7268 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:53:30.020855 7363 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:53:30.020901 7268 server_base.cc:1034] running on GCE node
W20260929 17:53:30.020869 7362 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:53:30.020866 7365 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:53:30.021667 7268 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:53:30.022253 7268 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:53:30.023406 7268 hybrid_clock.cc:648] HybridClock initialized: now 1790704410023381 us; error 33 us; skew 500 ppm
Sep 29 17:53:30 dist-test-slave-53nh krb5kdc[6478](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704410, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:53:30.026000 7268 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260929 17:53:30.026882 7268 webserver.cc:460] Webserver started at http://127.0.98.1:41565/ using document root <none> and password file <none>
I20260929 17:53:30.027410 7268 fs_manager.cc:359] Metadata directory not provided
I20260929 17:53:30.027486 7268 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:53:30.031153 7268 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.002s
I20260929 17:53:30.033070 7372 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:53:30.034264 7268 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 17:53:30.034363 7268 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704375975685-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704375975685-392-0/minicluster-data/ts-0/wal
uuid: "7cb83ff6a04043169a3cf2e232462596"
format_stamp: "Formatted at 2026-09-29 17:53:28 on dist-test-slave-53nh"
server_key: "3ac0901fb019f271eff2471049c6f14a"
server_key_iv: "94db964e8713f1ec1a0f1698e158718f"
server_key_version: "encryptionkey@0"
I20260929 17:53:30.034881 7268 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704375975685-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704375975685-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704375975685-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:53:30.045255 7268 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:53:30.048296 7268 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:53:30.048456 7268 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:53:30.049044 7268 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 17:53:30.049934 7268 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:53:30.049981 7268 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:53:30.050019 7268 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:53:30.050037 7268 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 17:53:30.058898 7268 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46193
I20260929 17:53:30.058916 7485 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46193 every 8 connection(s)
I20260929 17:53:30.059882 7268 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704375975685-392-0/minicluster-data/ts-0/data/info.pb
I20260929 17:53:30.060281 6848 heartbeater.cc:503] Master 127.0.98.62:39863 was elected leader, sending a full tablet report...
I20260929 17:53:30.066939 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskup3Soe/build/debug/bin/kudu as pid 7268
I20260929 17:53:30.067112 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskup3Soe/build/debug/bin/kudu with pid 6719
Sep 29 17:53:30 dist-test-slave-53nh krb5kdc[6478](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704410, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 29 17:53:30 dist-test-slave-53nh krb5kdc[6478](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704410, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Sep 29 17:53:30 dist-test-slave-53nh krb5kdc[6478](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704410, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260929 17:53:30.074055 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskup3Soe/build/debug/bin/kudu
/tmp/dist-test-taskup3Soe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704375975685-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704375975685-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704375975685-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704375975685-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:43951
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:39863
--webserver_port=33415
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704375975685-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:39863,127.0.98.61:34183
--tserver_master_addrs=127.0.98.62:39863,127.0.98.61:34183,127.0.98.60:39371 with env {KRB5CCNAME=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704375975685-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704375975685-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704375975685-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 17:53:30.074591 6562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:30.062304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51811 (local address 127.0.98.62:39863)
0929 17:53:30.062444 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:30.062448 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:30.063330 (+ 882us) server_negotiation.cc:407] Connection header received
0929 17:53:30.063852 (+ 522us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:30.063855 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:30.063905 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:30.063969 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:53:30.065295 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:30.066073 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:30.067346 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:30.067618 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:30.067682 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:30.070606 (+ 2924us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:53:30.070610 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:53:30.070613 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:53:30.070635 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:53:30.072405 (+ 1770us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:53:30.072927 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:53:30.072931 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:53:30.072932 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:53:30.073053 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:53:30.073409 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:53:30.073413 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:53:30.073414 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:53:30.073608 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:53:30.073676 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0929 17:53:30.074325 (+ 649us) server_negotiation.cc:299] Negotiation successful
0929 17:53:30.074439 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260929 17:53:30.075965 7245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:30.063040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58497 (local address 127.0.98.61:34183)
0929 17:53:30.063162 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:30.063167 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:30.063182 (+ 15us) server_negotiation.cc:407] Connection header received
0929 17:53:30.063890 (+ 708us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:30.063894 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:30.063950 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:30.064001 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:53:30.065267 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:30.066073 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:30.069750 (+ 3677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:30.070091 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:30.070168 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:30.072561 (+ 2393us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:53:30.072565 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:53:30.072568 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:53:30.072596 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:53:30.074329 (+ 1733us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:53:30.074772 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:53:30.074775 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:53:30.074777 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:53:30.074850 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:53:30.075132 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:53:30.075135 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:53:30.075137 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:53:30.075330 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:53:30.075398 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0929 17:53:30.075719 (+ 321us) server_negotiation.cc:299] Negotiation successful
0929 17:53:30.075834 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260929 17:53:30.076138 7490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:30.062117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39371 (local address 127.0.98.1:35457)
0929 17:53:30.063072 (+ 955us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:30.063084 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 17:53:30.063663 (+ 579us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:53:30.064150 (+ 487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:53:30.064158 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:53:30.064582 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:53:30.065142 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:30.065147 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:30.066104 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:30.066107 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:30.067924 (+ 1817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:30.067934 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:30.068585 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:30.068588 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:30.068672 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:30.069579 (+ 907us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:53:30.069593 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:53:30.070936 (+ 1343us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:53:30.073927 (+ 2991us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:53:30.073930 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:53:30.073932 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:53:30.074156 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:53:30.074443 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:53:30.074445 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:53:30.074447 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:53:30.074553 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:53:30.074963 (+ 410us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:53:30.074966 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:53:30.075045 (+ 79us) client_negotiation.cc:769] Sending connection context
0929 17:53:30.075130 (+ 85us) client_negotiation.cc:240] Negotiation successful
0929 17:53:30.075262 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":882,"mutex_wait_us":251,"spinlock_wait_cycles":11008,"thread_start_us":103,"threads_started":1}
I20260929 17:53:30.076400 7493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:30.062117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39863 (local address 127.0.98.1:51811)
0929 17:53:30.063247 (+ 1130us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:30.063259 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 17:53:30.063660 (+ 401us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:53:30.064149 (+ 489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:53:30.064158 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:53:30.064578 (+ 420us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:53:30.065139 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:30.065147 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:30.066213 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:30.066216 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:30.067194 (+ 978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:30.067203 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:30.067857 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:30.067861 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:30.068035 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:30.068643 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:53:30.068661 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:53:30.070438 (+ 1777us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:53:30.072523 (+ 2085us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:53:30.072529 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:53:30.072544 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:53:30.072846 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:53:30.073128 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:53:30.073131 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:53:30.073134 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:53:30.073307 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:53:30.073752 (+ 445us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:53:30.073758 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:53:30.074045 (+ 287us) client_negotiation.cc:769] Sending connection context
0929 17:53:30.074729 (+ 684us) client_negotiation.cc:240] Negotiation successful
0929 17:53:30.075051 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1042,"mutex_wait_us":468,"spinlock_wait_cycles":26240,"thread_start_us":84,"threads_started":1}
I20260929 17:53:30.076406 7491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:30.062117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34183 (local address 127.0.98.1:58497)
0929 17:53:30.062779 (+ 662us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:30.062829 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0929 17:53:30.063658 (+ 829us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:53:30.064149 (+ 491us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:53:30.064157 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:53:30.064576 (+ 419us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:53:30.065129 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:30.065143 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:30.066171 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:30.066174 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:30.069599 (+ 3425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:30.069608 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:30.070313 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:30.070316 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:30.070410 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:30.071070 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:53:30.071090 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:53:30.072367 (+ 1277us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:53:30.074435 (+ 2068us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:53:30.074438 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:53:30.074440 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:53:30.074665 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:53:30.074958 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:53:30.074960 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:53:30.074962 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:53:30.075035 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:53:30.075463 (+ 428us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:53:30.075466 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:53:30.075546 (+ 80us) client_negotiation.cc:769] Sending connection context
0929 17:53:30.076141 (+ 595us) client_negotiation.cc:240] Negotiation successful
0929 17:53:30.076263 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":457,"mutex_wait_us":88,"thread_start_us":132,"threads_started":1}
I20260929 17:53:30.077142 7492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:30.062294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35457 (local address 127.0.98.60:39371)
0929 17:53:30.063305 (+ 1011us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:30.063319 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:30.063334 (+ 15us) server_negotiation.cc:407] Connection header received
0929 17:53:30.063942 (+ 608us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:30.063945 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:30.063990 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:30.064098 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:53:30.065281 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:30.065974 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:30.068046 (+ 2072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:30.068383 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:30.068471 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:30.071083 (+ 2612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:53:30.071087 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:53:30.071090 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:53:30.071112 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:53:30.073811 (+ 2699us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:53:30.074275 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:53:30.074278 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:53:30.074280 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:53:30.074358 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:53:30.074652 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:53:30.074655 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:53:30.074657 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:53:30.074855 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:53:30.074929 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0929 17:53:30.076824 (+ 1895us) server_negotiation.cc:299] Negotiation successful
0929 17:53:30.076942 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":930,"thread_start_us":81,"threads_started":1}
I20260929 17:53:30.077462 7487 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39863
I20260929 17:53:30.077715 7487 heartbeater.cc:464] Registering TS with master...
I20260929 17:53:30.077816 7488 heartbeater.cc:347] Connected to a master server at 127.0.98.61:34183
I20260929 17:53:30.077816 7486 heartbeater.cc:347] Connected to a master server at 127.0.98.60:39371
I20260929 17:53:30.077955 7486 heartbeater.cc:464] Registering TS with master...
I20260929 17:53:30.078127 7488 heartbeater.cc:464] Registering TS with master...
I20260929 17:53:30.078271 7487 heartbeater.cc:511] Master 127.0.98.62:39863 requested a full tablet report, sending...
I20260929 17:53:30.078269 7486 heartbeater.cc:511] Master 127.0.98.60:39371 requested a full tablet report, sending...
I20260929 17:53:30.078269 7488 heartbeater.cc:511] Master 127.0.98.61:34183 requested a full tablet report, sending...
I20260929 17:53:30.079635 6523 ts_manager.cc:194] Re-registered known tserver with Master: 7cb83ff6a04043169a3cf2e232462596 (127.0.98.1:46193)
I20260929 17:53:30.079769 7307 ts_manager.cc:194] Registered new tserver with Master: 7cb83ff6a04043169a3cf2e232462596 (127.0.98.1:46193)
I20260929 17:53:30.079767 7123 ts_manager.cc:194] Registered new tserver with Master: 7cb83ff6a04043169a3cf2e232462596 (127.0.98.1:46193)
I20260929 17:53:30.080277 6523 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:51811
W20260929 17:53:30.186422 7494 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:53:30.186690 7494 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:53:30.186733 7494 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:53:30.190781 7494 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:53:30.190851 7494 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:53:30.190912 7494 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 17:53:30.196373 7494 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704375975685-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704375975685-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704375975685-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:43951
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704375975685-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=33415
--tserver_master_addrs=127.0.98.62:39863,127.0.98.61:34183,127.0.98.60:39371
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7494
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704375975685-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:53:30.197525 7494 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:53:30.198603 7494 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:53:30.208220 7499 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:53:30.208223 7502 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:53:30.208298 7494 server_base.cc:1034] running on GCE node
W20260929 17:53:30.208220 7500 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:53:30.208977 7494 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:53:30.209656 7494 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:53:30.210832 7494 hybrid_clock.cc:648] HybridClock initialized: now 1790704410210800 us; error 46 us; skew 500 ppm
Sep 29 17:53:30 dist-test-slave-53nh krb5kdc[6478](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704410, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:53:30.213682 7494 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260929 17:53:30.214531 7494 webserver.cc:460] Webserver started at http://127.0.98.2:33415/ using document root <none> and password file <none>
I20260929 17:53:30.215058 7494 fs_manager.cc:359] Metadata directory not provided
I20260929 17:53:30.215127 7494 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:53:30.218935 7494 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260929 17:53:30.220901 7509 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:53:30.222143 7494 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 17:53:30.222275 7494 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704375975685-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704375975685-392-0/minicluster-data/ts-1/wal
uuid: "ade19fb6904045318b25a43744e7c3f3"
format_stamp: "Formatted at 2026-09-29 17:53:28 on dist-test-slave-53nh"
server_key: "fc81c3bac5f6f6f84b78f8fbd77d8000"
server_key_iv: "1c42630f4150badf894fe6361a951c14"
server_key_version: "encryptionkey@0"
I20260929 17:53:30.222821 7494 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704375975685-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704375975685-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704375975685-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:53:30.243649 7494 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:53:30.246865 7494 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:53:30.247032 7494 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:53:30.247700 7494 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 17:53:30.248534 7494 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:53:30.248579 7494 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:53:30.248617 7494 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:53:30.248643 7494 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 17:53:30.257087 7494 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43951
I20260929 17:53:30.257110 7622 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43951 every 8 connection(s)
I20260929 17:53:30.258157 7494 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704375975685-392-0/minicluster-data/ts-1/data/info.pb
I20260929 17:53:30.264586 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskup3Soe/build/debug/bin/kudu as pid 7494
I20260929 17:53:30.264777 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskup3Soe/build/debug/bin/kudu with pid 6853
Sep 29 17:53:30 dist-test-slave-53nh krb5kdc[6478](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704410, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Sep 29 17:53:30 dist-test-slave-53nh krb5kdc[6478](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704410, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 29 17:53:30 dist-test-slave-53nh krb5kdc[6478](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704410, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260929 17:53:30.272946 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskup3Soe/build/debug/bin/kudu
/tmp/dist-test-taskup3Soe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704375975685-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704375975685-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704375975685-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704375975685-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:41685
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:39863
--webserver_port=43769
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704375975685-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:39863,127.0.98.61:34183
--tserver_master_addrs=127.0.98.62:39863,127.0.98.61:34183,127.0.98.60:39371 with env {KRB5CCNAME=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704375975685-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704375975685-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704375975685-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 17:53:30.274710 7245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:30.259919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58343 (local address 127.0.98.61:34183)
0929 17:53:30.260051 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:30.260055 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:30.261236 (+ 1181us) server_negotiation.cc:407] Connection header received
0929 17:53:30.262671 (+ 1435us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:30.262675 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:30.262722 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:30.262807 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:53:30.263955 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:30.264797 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:30.266094 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:30.266460 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:30.266557 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:30.270486 (+ 3929us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:53:30.270490 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:53:30.270492 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:53:30.270521 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:53:30.271932 (+ 1411us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:53:30.272664 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:53:30.272668 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:53:30.272670 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:53:30.272752 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:53:30.273208 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:53:30.273211 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:53:30.273213 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:53:30.273643 (+ 430us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:53:30.273723 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0929 17:53:30.274424 (+ 701us) server_negotiation.cc:299] Negotiation successful
0929 17:53:30.274548 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 17:53:30.274710 7492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:30.260042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39681 (local address 127.0.98.60:39371)
0929 17:53:30.260162 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:30.260167 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:30.261473 (+ 1306us) server_negotiation.cc:407] Connection header received
0929 17:53:30.262672 (+ 1199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:30.262675 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:30.262725 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:30.262805 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:53:30.263990 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:30.264797 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:30.266966 (+ 2169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:30.267561 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:30.267646 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:30.270541 (+ 2895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:53:30.270544 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:53:30.270547 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:53:30.270575 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:53:30.272093 (+ 1518us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:53:30.272664 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:53:30.272668 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:53:30.272670 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:53:30.272752 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:53:30.273208 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:53:30.273211 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:53:30.273213 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:53:30.273425 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:53:30.273489 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0929 17:53:30.274424 (+ 935us) server_negotiation.cc:299] Negotiation successful
0929 17:53:30.274548 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 17:53:30.274718 6562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:30.260657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58103 (local address 127.0.98.62:39863)
0929 17:53:30.260779 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:30.260784 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:30.261561 (+ 777us) server_negotiation.cc:407] Connection header received
0929 17:53:30.262431 (+ 870us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:30.262434 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:30.262479 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:30.262550 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:53:30.263984 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:30.264603 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:30.267059 (+ 2456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:30.267402 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:30.267479 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:30.270174 (+ 2695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:53:30.270178 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:53:30.270181 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:53:30.270211 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:53:30.272526 (+ 2315us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:53:30.273020 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:53:30.273025 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:53:30.273027 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:53:30.273117 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:53:30.273615 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:53:30.273618 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:53:30.273620 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:53:30.273834 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:53:30.273908 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0929 17:53:30.274437 (+ 529us) server_negotiation.cc:299] Negotiation successful
0929 17:53:30.274562 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 17:53:30.276455 7628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:30.260318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39371 (local address 127.0.98.2:39681)
0929 17:53:30.261377 (+ 1059us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:30.261390 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 17:53:30.262227 (+ 837us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:53:30.262893 (+ 666us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:53:30.262896 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:53:30.263257 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:53:30.263812 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:30.263832 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:30.264928 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:30.264931 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:30.266836 (+ 1905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:30.266845 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:30.267825 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:30.267829 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:30.268168 (+ 339us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:30.268594 (+ 426us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:53:30.268607 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:53:30.270411 (+ 1804us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:53:30.272199 (+ 1788us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:53:30.272202 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:53:30.272204 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:53:30.272550 (+ 346us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:53:30.272858 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:53:30.272860 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:53:30.272862 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:53:30.273100 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:53:30.273781 (+ 681us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:53:30.273783 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:53:30.274133 (+ 350us) client_negotiation.cc:769] Sending connection context
0929 17:53:30.274955 (+ 822us) client_negotiation.cc:240] Negotiation successful
0929 17:53:30.275319 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":981,"mutex_wait_us":1155,"spinlock_wait_cycles":73856,"thread_start_us":119,"threads_started":1}
I20260929 17:53:30.276700 7627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:30.260317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34183 (local address 127.0.98.2:58343)
0929 17:53:30.261082 (+ 765us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:30.261139 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0929 17:53:30.262560 (+ 1421us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:53:30.262827 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:53:30.262830 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:53:30.263256 (+ 426us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:53:30.263816 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:30.263831 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:30.264987 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:30.264989 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:30.265977 (+ 988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:30.265987 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:30.266730 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:30.266734 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:30.267015 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:30.267607 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:53:30.267629 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:53:30.270325 (+ 2696us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:53:30.272071 (+ 1746us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:53:30.272078 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:53:30.272096 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:53:30.272550 (+ 454us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:53:30.272858 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:53:30.272860 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:53:30.272862 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:53:30.273100 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:53:30.273746 (+ 646us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:53:30.273753 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:53:30.274133 (+ 380us) client_negotiation.cc:769] Sending connection context
0929 17:53:30.274867 (+ 734us) client_negotiation.cc:240] Negotiation successful
0929 17:53:30.275376 (+ 509us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":530,"mutex_wait_us":327,"spinlock_wait_cycles":18560,"thread_start_us":141,"threads_started":1}
I20260929 17:53:30.276465 7629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:30.261074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39863 (local address 127.0.98.2:58103)
0929 17:53:30.261483 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:30.261493 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 17:53:30.262241 (+ 748us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:53:30.262728 (+ 487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:53:30.262735 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:53:30.263261 (+ 526us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:53:30.263812 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:30.263832 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:30.264858 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:30.264861 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:30.266923 (+ 2062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:30.266934 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:30.267623 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:30.267626 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:30.267720 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:30.268156 (+ 436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:53:30.268169 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:53:30.269989 (+ 1820us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:53:30.272648 (+ 2659us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:53:30.272651 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:53:30.272653 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:53:30.272884 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:53:30.273437 (+ 553us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:53:30.273439 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:53:30.273441 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:53:30.273521 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:53:30.273961 (+ 440us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:53:30.273964 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:53:30.274133 (+ 169us) client_negotiation.cc:769] Sending connection context
0929 17:53:30.275387 (+ 1254us) client_negotiation.cc:240] Negotiation successful
0929 17:53:30.275535 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":350,"mutex_wait_us":1041,"spinlock_wait_cycles":26112,"thread_start_us":89,"threads_started":1}
I20260929 17:53:30.277930 7623 heartbeater.cc:347] Connected to a master server at 127.0.98.60:39371
I20260929 17:53:30.277930 7624 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39863
I20260929 17:53:30.278234 7625 heartbeater.cc:347] Connected to a master server at 127.0.98.61:34183
I20260929 17:53:30.278204 7623 heartbeater.cc:464] Registering TS with master...
I20260929 17:53:30.278368 7625 heartbeater.cc:464] Registering TS with master...
I20260929 17:53:30.278204 7624 heartbeater.cc:464] Registering TS with master...
I20260929 17:53:30.278895 7624 heartbeater.cc:511] Master 127.0.98.62:39863 requested a full tablet report, sending...
I20260929 17:53:30.279522 7623 heartbeater.cc:511] Master 127.0.98.60:39371 requested a full tablet report, sending...
I20260929 17:53:30.280088 6523 ts_manager.cc:194] Re-registered known tserver with Master: ade19fb6904045318b25a43744e7c3f3 (127.0.98.2:43951)
I20260929 17:53:30.280181 7307 ts_manager.cc:194] Registered new tserver with Master: ade19fb6904045318b25a43744e7c3f3 (127.0.98.2:43951)
I20260929 17:53:30.280224 7625 heartbeater.cc:511] Master 127.0.98.61:34183 requested a full tablet report, sending...
I20260929 17:53:30.280793 6523 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:58103
I20260929 17:53:30.280947 7123 ts_manager.cc:194] Registered new tserver with Master: ade19fb6904045318b25a43744e7c3f3 (127.0.98.2:43951)
W20260929 17:53:30.382898 7630 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:53:30.383154 7630 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:53:30.383203 7630 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:53:30.387153 7630 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:53:30.387214 7630 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:53:30.387288 7630 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 17:53:30.392604 7630 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704375975685-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704375975685-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704375975685-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:41685
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704375975685-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=43769
--tserver_master_addrs=127.0.98.62:39863,127.0.98.61:34183,127.0.98.60:39371
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7630
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704375975685-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:53:30.393721 7630 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:53:30.394737 7630 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:53:30.404202 7639 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:53:30.404220 7636 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:53:30.404217 7637 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:53:30.404778 7630 server_base.cc:1034] running on GCE node
I20260929 17:53:30.405206 7630 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:53:30.405826 7630 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:53:30.406987 7630 hybrid_clock.cc:648] HybridClock initialized: now 1790704410406967 us; error 31 us; skew 500 ppm
Sep 29 17:53:30 dist-test-slave-53nh krb5kdc[6478](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704410, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:53:30.409617 7630 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260929 17:53:30.410472 7630 webserver.cc:460] Webserver started at http://127.0.98.3:43769/ using document root <none> and password file <none>
I20260929 17:53:30.411012 7630 fs_manager.cc:359] Metadata directory not provided
I20260929 17:53:30.411090 7630 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:53:30.414942 7630 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 17:53:30.417076 7646 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:53:30.418342 7630 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 17:53:30.418449 7630 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704375975685-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704375975685-392-0/minicluster-data/ts-2/wal
uuid: "c4d0d8b2d8994344a6ec327d5c389bba"
format_stamp: "Formatted at 2026-09-29 17:53:29 on dist-test-slave-53nh"
server_key: "d1ad7677298cd0e0e27082c0e2868c8a"
server_key_iv: "156845e626864f41fc4bc5d0920d7255"
server_key_version: "encryptionkey@0"
I20260929 17:53:30.418989 7630 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704375975685-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704375975685-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704375975685-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:53:30.431362 7651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:30.427934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39371 (local address 127.0.0.1:46826)
0929 17:53:30.428156 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:30.428171 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 17:53:30.428304 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:53:30.428566 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:53:30.428569 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:53:30.428586 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:53:30.428688 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:30.428693 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:30.429522 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:30.429525 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:30.430301 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:30.430311 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:30.430965 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:30.430969 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:30.431057 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:30.431080 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 17:53:30.431134 (+ 54us) client_negotiation.cc:240] Negotiation successful
0929 17:53:30.431212 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":63,"threads_started":1}
I20260929 17:53:30.431353 7492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:30.428018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46826 (local address 127.0.98.60:39371)
0929 17:53:30.428157 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:30.428161 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:30.428247 (+ 86us) server_negotiation.cc:407] Connection header received
0929 17:53:30.428415 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:30.428418 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:30.428470 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:30.428545 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:53:30.428801 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:30.429412 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:30.430419 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:30.430854 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:30.430948 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:30.431117 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0929 17:53:30.431161 (+ 44us) server_negotiation.cc:299] Negotiation successful
0929 17:53:30.431224 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 17:53:30.432304 7308 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 60abdd24a53146259072b5a6b7220ec0 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260929 17:53:30.432719 7346 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60abdd24a53146259072b5a6b7220ec0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 14cabac4c134442b901728b1c630deb5. Latest consensus state: current_term: 1 leader_uuid: "14cabac4c134442b901728b1c630deb5" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "14cabac4c134442b901728b1c630deb5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39863 } } peers { permanent_uuid: "952a9b1a252747a8acdcfa8e04debe30" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34183 } attrs { promote: false } } peers { permanent_uuid: "60abdd24a53146259072b5a6b7220ec0" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39371 } attrs { promote: true } } }
I20260929 17:53:30.432859 7346 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60abdd24a53146259072b5a6b7220ec0 [sys.catalog]: This master's current role is: LEARNER
I20260929 17:53:30.452896 7630 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:53:30.455806 7630 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:53:30.455955 7630 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:53:30.456506 7630 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 17:53:30.457399 7630 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:53:30.457445 7630 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:53:30.457482 7630 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:53:30.457508 7630 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 17:53:30.465675 7630 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41685
I20260929 17:53:30.465715 7760 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41685 every 8 connection(s)
I20260929 17:53:30.466672 7630 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704375975685-392-0/minicluster-data/ts-2/data/info.pb
I20260929 17:53:30.469563 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskup3Soe/build/debug/bin/kudu as pid 7630
Sep 29 17:53:30 dist-test-slave-53nh krb5kdc[6478](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704410, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Sep 29 17:53:30 dist-test-slave-53nh krb5kdc[6478](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704410, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 29 17:53:30 dist-test-slave-53nh krb5kdc[6478](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704410, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260929 17:53:30.483359 7492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:30.468489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39811 (local address 127.0.98.60:39371)
0929 17:53:30.468605 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:30.468610 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:30.470310 (+ 1700us) server_negotiation.cc:407] Connection header received
0929 17:53:30.471387 (+ 1077us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:30.471391 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:30.471450 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:30.471520 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:53:30.473408 (+ 1888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:30.474200 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:30.475834 (+ 1634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:30.476153 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:30.476239 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:30.479375 (+ 3136us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:53:30.479379 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:53:30.479382 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:53:30.479410 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:53:30.481034 (+ 1624us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:53:30.481588 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:53:30.481591 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:53:30.481593 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:53:30.481664 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:53:30.482096 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:53:30.482100 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:53:30.482101 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:53:30.482317 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:53:30.482407 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0929 17:53:30.483072 (+ 665us) server_negotiation.cc:299] Negotiation successful
0929 17:53:30.483186 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":36}
I20260929 17:53:30.484236 6562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:30.468818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53027 (local address 127.0.98.62:39863)
0929 17:53:30.468920 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:30.468924 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:30.471445 (+ 2521us) server_negotiation.cc:407] Connection header received
0929 17:53:30.471542 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:30.471545 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:30.471598 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:30.471675 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:53:30.473484 (+ 1809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:30.474220 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:30.478129 (+ 3909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:30.478465 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:30.478540 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:30.480725 (+ 2185us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:53:30.480728 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:53:30.480731 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:53:30.480754 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:53:30.482490 (+ 1736us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:53:30.482960 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:53:30.482963 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:53:30.482964 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:53:30.483047 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:53:30.483331 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:53:30.483335 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:53:30.483336 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:53:30.483534 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:53:30.483616 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0929 17:53:30.483966 (+ 350us) server_negotiation.cc:299] Negotiation successful
0929 17:53:30.484085 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260929 17:53:30.484967 7766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:30.469177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39863 (local address 127.0.98.3:53027)
0929 17:53:30.471372 (+ 2195us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:30.471384 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 17:53:30.471467 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:53:30.471818 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:53:30.471828 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:53:30.472497 (+ 669us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:53:30.473377 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:30.473383 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:30.474362 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:30.474365 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:30.477989 (+ 3624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:30.478000 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:30.478684 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:30.478687 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:30.478787 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:30.479190 (+ 403us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:53:30.479202 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:53:30.480599 (+ 1397us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:53:30.482604 (+ 2005us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:53:30.482607 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:53:30.482609 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:53:30.482848 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:53:30.483148 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:53:30.483151 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:53:30.483153 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:53:30.483230 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:53:30.483672 (+ 442us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:53:30.483675 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:53:30.483766 (+ 91us) client_negotiation.cc:769] Sending connection context
0929 17:53:30.484370 (+ 604us) client_negotiation.cc:240] Negotiation successful
0929 17:53:30.484507 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2137,"mutex_wait_us":1326,"spinlock_wait_cycles":52352,"thread_start_us":132,"threads_started":1}
I20260929 17:53:30.484970 7765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:30.469104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39371 (local address 127.0.98.3:39811)
0929 17:53:30.470167 (+ 1063us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:30.470216 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 17:53:30.471049 (+ 833us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:53:30.471819 (+ 770us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:53:30.471828 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:53:30.472435 (+ 607us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:53:30.473250 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:30.473270 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:30.474362 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:30.474365 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:30.475697 (+ 1332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:30.475706 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:30.476384 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:30.476388 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:30.476585 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:30.477131 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:53:30.477152 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:53:30.479146 (+ 1994us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:53:30.481134 (+ 1988us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:53:30.481140 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:53:30.481157 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:53:30.481490 (+ 333us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:53:30.481781 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:53:30.481784 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:53:30.481786 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:53:30.481966 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:53:30.482459 (+ 493us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:53:30.482467 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:53:30.482796 (+ 329us) client_negotiation.cc:769] Sending connection context
0929 17:53:30.483518 (+ 722us) client_negotiation.cc:240] Negotiation successful
0929 17:53:30.483882 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":886,"mutex_wait_us":74,"spinlock_wait_cycles":51328,"thread_start_us":107,"threads_started":1}
I20260929 17:53:30.485935 7773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:30.471284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40090 (local address 127.0.98.62:39863)
0929 17:53:30.471810 (+ 526us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:30.471814 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:30.472108 (+ 294us) server_negotiation.cc:407] Connection header received
0929 17:53:30.472147 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:30.472149 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:30.472200 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:30.472270 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:53:30.473174 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:30.473994 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:30.475121 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:30.475445 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:30.475531 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:30.481553 (+ 6022us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:53:30.481557 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:53:30.481559 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:53:30.481582 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:53:30.484061 (+ 2479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:53:30.484589 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:53:30.484592 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:53:30.484594 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:53:30.484675 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:53:30.484940 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:53:30.484943 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:53:30.484944 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:53:30.485158 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:53:30.485287 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0929 17:53:30.485604 (+ 317us) server_negotiation.cc:299] Negotiation successful
0929 17:53:30.485748 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":463,"thread_start_us":79,"threads_started":1}
I20260929 17:53:30.486585 7761 heartbeater.cc:347] Connected to a master server at 127.0.98.60:39371
I20260929 17:53:30.486842 7761 heartbeater.cc:464] Registering TS with master...
I20260929 17:53:30.487402 7761 heartbeater.cc:511] Master 127.0.98.60:39371 requested a full tablet report, sending...
I20260929 17:53:30.488112 7776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:30.472207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41502 (local address 127.0.98.61:34183)
0929 17:53:30.472719 (+ 512us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:30.472723 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:30.474357 (+ 1634us) server_negotiation.cc:407] Connection header received
0929 17:53:30.474396 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:30.474398 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:30.474442 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:30.474512 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:53:30.475291 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:30.475986 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:30.477101 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:30.477516 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:30.477642 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:30.479715 (+ 2073us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:53:30.479719 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:53:30.479721 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:53:30.479748 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:53:30.482876 (+ 3128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:53:30.486593 (+ 3717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:53:30.486596 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:53:30.486598 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:53:30.486681 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:53:30.486985 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:53:30.486989 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:53:30.486990 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:53:30.487189 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:53:30.487303 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0929 17:53:30.487843 (+ 540us) server_negotiation.cc:299] Negotiation successful
0929 17:53:30.487965 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":429,"thread_start_us":85,"threads_started":1}
I20260929 17:53:30.488605 7307 ts_manager.cc:194] Registered new tserver with Master: c4d0d8b2d8994344a6ec327d5c389bba (127.0.98.3:41685)
I20260929 17:53:30.488986 7762 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39863
I20260929 17:53:30.489087 7762 heartbeater.cc:464] Registering TS with master...
I20260929 17:53:30.489251 7762 heartbeater.cc:511] Master 127.0.98.62:39863 requested a full tablet report, sending...
I20260929 17:53:30.489537 7772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:30.471283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46830 (local address 127.0.98.60:39371)
0929 17:53:30.473192 (+ 1909us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:30.473204 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:30.473216 (+ 12us) server_negotiation.cc:407] Connection header received
0929 17:53:30.473248 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:30.473250 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:30.473288 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:30.473358 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:53:30.475953 (+ 2595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:30.476766 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:30.477834 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:30.478145 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:30.478247 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:30.480669 (+ 2422us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:53:30.480673 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:53:30.480676 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:53:30.480700 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:53:30.487525 (+ 6825us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:53:30.487964 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:53:30.487968 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:53:30.487969 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:53:30.488049 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:53:30.488355 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:53:30.488359 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:53:30.488360 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:53:30.488581 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:53:30.488701 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0929 17:53:30.489275 (+ 574us) server_negotiation.cc:299] Negotiation successful
0929 17:53:30.489389 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1843,"thread_start_us":78,"threads_started":1}
I20260929 17:53:30.489903 6522 ts_manager.cc:194] Re-registered known tserver with Master: c4d0d8b2d8994344a6ec327d5c389bba (127.0.98.3:41685)
I20260929 17:53:30.490634 6522 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:53027
I20260929 17:53:30.491881 7767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:30.469105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34183 (local address 127.0.98.3:50387)
0929 17:53:30.470166 (+ 1061us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:30.470207 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 17:53:30.471040 (+ 833us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:53:30.472032 (+ 992us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:53:30.472034 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:53:30.472436 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:53:30.473275 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:30.473290 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:30.476228 (+ 2938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:30.476231 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:30.480527 (+ 4296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:30.480538 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:30.481208 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:30.481212 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:30.481311 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:30.481898 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:53:30.481913 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:53:30.483247 (+ 1334us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:53:30.490317 (+ 7070us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:53:30.490320 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:53:30.490322 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:53:30.490556 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:53:30.490830 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:53:30.490832 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:53:30.490834 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:53:30.490922 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:53:30.491340 (+ 418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:53:30.491344 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:53:30.491471 (+ 127us) client_negotiation.cc:769] Sending connection context
0929 17:53:30.491566 (+ 95us) client_negotiation.cc:240] Negotiation successful
0929 17:53:30.491696 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":896,"mutex_wait_us":581,"spinlock_wait_cycles":8704,"thread_start_us":161,"threads_started":1}
I20260929 17:53:30.491932 7245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:30.468615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50387 (local address 127.0.98.61:34183)
0929 17:53:30.468732 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:30.468737 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:30.470310 (+ 1573us) server_negotiation.cc:407] Connection header received
0929 17:53:30.471603 (+ 1293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:30.471606 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:30.471656 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:30.471706 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:53:30.474433 (+ 2727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:30.475169 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:30.480655 (+ 5486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:30.480960 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:30.481059 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:30.483386 (+ 2327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:53:30.483389 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:53:30.483392 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:53:30.483414 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:53:30.485390 (+ 1976us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:53:30.490644 (+ 5254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:53:30.490647 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:53:30.490649 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:53:30.490736 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:53:30.491029 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:53:30.491032 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:53:30.491034 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:53:30.491239 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:53:30.491332 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0929 17:53:30.491648 (+ 316us) server_negotiation.cc:299] Negotiation successful
0929 17:53:30.491798 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260929 17:53:30.489204 6523 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40090:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 17:53:30.492965 6523 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 17:53:30.494433 7763 heartbeater.cc:347] Connected to a master server at 127.0.98.61:34183
I20260929 17:53:30.494524 7763 heartbeater.cc:464] Registering TS with master...
I20260929 17:53:30.494696 7763 heartbeater.cc:511] Master 127.0.98.61:34183 requested a full tablet report, sending...
I20260929 17:53:30.495376 7123 ts_manager.cc:194] Registered new tserver with Master: c4d0d8b2d8994344a6ec327d5c389bba (127.0.98.3:41685)
I20260929 17:53:30.495636 7779 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14cabac4c134442b901728b1c630deb5 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "14cabac4c134442b901728b1c630deb5" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "14cabac4c134442b901728b1c630deb5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39863 } } peers { permanent_uuid: "952a9b1a252747a8acdcfa8e04debe30" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34183 } attrs { promote: false } } peers { permanent_uuid: "60abdd24a53146259072b5a6b7220ec0" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39371 } attrs { promote: true } } }
I20260929 17:53:30.495757 7779 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14cabac4c134442b901728b1c630deb5 [sys.catalog]: This master's current role is: LEADER
I20260929 17:53:30.498544 7780 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 14cabac4c134442b901728b1c630deb5: attempting to promote NON_VOTER 60abdd24a53146259072b5a6b7220ec0 to VOTER
I20260929 17:53:30.498869 7780 consensus_queue.cc:237] T 00000000000000000000000000000000 P 14cabac4c134442b901728b1c630deb5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "14cabac4c134442b901728b1c630deb5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39863 } } peers { permanent_uuid: "952a9b1a252747a8acdcfa8e04debe30" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34183 } attrs { promote: false } } peers { permanent_uuid: "60abdd24a53146259072b5a6b7220ec0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39371 } attrs { promote: false } }
I20260929 17:53:30.499907 7133 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 952a9b1a252747a8acdcfa8e04debe30 [term 1 FOLLOWER]: Refusing update from remote peer 14cabac4c134442b901728b1c630deb5: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260929 17:53:30.500046 7309 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 60abdd24a53146259072b5a6b7220ec0 [term 1 LEARNER]: Refusing update from remote peer 14cabac4c134442b901728b1c630deb5: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260929 17:53:30.500334 7779 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 14cabac4c134442b901728b1c630deb5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "952a9b1a252747a8acdcfa8e04debe30" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34183 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260929 17:53:30.500460 7634 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 14cabac4c134442b901728b1c630deb5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60abdd24a53146259072b5a6b7220ec0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39371 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20260929 17:53:30.502501 7634 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 14cabac4c134442b901728b1c630deb5 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 60abdd24a53146259072b5a6b7220ec0 (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "14cabac4c134442b901728b1c630deb5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39863 } } peers { permanent_uuid: "952a9b1a252747a8acdcfa8e04debe30" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34183 } attrs { promote: false } } peers { permanent_uuid: "60abdd24a53146259072b5a6b7220ec0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39371 } attrs { promote: false } } }
I20260929 17:53:30.503049 7133 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 952a9b1a252747a8acdcfa8e04debe30 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 60abdd24a53146259072b5a6b7220ec0 (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "14cabac4c134442b901728b1c630deb5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39863 } } peers { permanent_uuid: "952a9b1a252747a8acdcfa8e04debe30" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34183 } attrs { promote: false } } peers { permanent_uuid: "60abdd24a53146259072b5a6b7220ec0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39371 } attrs { promote: false } } }
I20260929 17:53:30.503165 7308 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 60abdd24a53146259072b5a6b7220ec0 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 60abdd24a53146259072b5a6b7220ec0 (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "14cabac4c134442b901728b1c630deb5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39863 } } peers { permanent_uuid: "952a9b1a252747a8acdcfa8e04debe30" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34183 } attrs { promote: false } } peers { permanent_uuid: "60abdd24a53146259072b5a6b7220ec0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39371 } attrs { promote: false } } }
I20260929 17:53:30.504024 7634 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14cabac4c134442b901728b1c630deb5 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "14cabac4c134442b901728b1c630deb5" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "14cabac4c134442b901728b1c630deb5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39863 } } peers { permanent_uuid: "952a9b1a252747a8acdcfa8e04debe30" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34183 } attrs { promote: false } } peers { permanent_uuid: "60abdd24a53146259072b5a6b7220ec0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39371 } attrs { promote: false } } }
I20260929 17:53:30.504117 7634 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14cabac4c134442b901728b1c630deb5 [sys.catalog]: This master's current role is: LEADER
I20260929 17:53:30.504235 7634 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14cabac4c134442b901728b1c630deb5 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "14cabac4c134442b901728b1c630deb5" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "14cabac4c134442b901728b1c630deb5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39863 } } peers { permanent_uuid: "952a9b1a252747a8acdcfa8e04debe30" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34183 } attrs { promote: false } } peers { permanent_uuid: "60abdd24a53146259072b5a6b7220ec0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39371 } attrs { promote: false } } }
I20260929 17:53:30.504313 7634 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14cabac4c134442b901728b1c630deb5 [sys.catalog]: This master's current role is: LEADER
I20260929 17:53:30.504343 7779 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14cabac4c134442b901728b1c630deb5 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "14cabac4c134442b901728b1c630deb5" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "14cabac4c134442b901728b1c630deb5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39863 } } peers { permanent_uuid: "952a9b1a252747a8acdcfa8e04debe30" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34183 } attrs { promote: false } } peers { permanent_uuid: "60abdd24a53146259072b5a6b7220ec0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39371 } attrs { promote: false } } }
I20260929 17:53:30.504422 7779 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14cabac4c134442b901728b1c630deb5 [sys.catalog]: This master's current role is: LEADER
I20260929 17:53:30.505132 7788 sys_catalog.cc:455] T 00000000000000000000000000000000 P 952a9b1a252747a8acdcfa8e04debe30 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "14cabac4c134442b901728b1c630deb5" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "14cabac4c134442b901728b1c630deb5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39863 } } peers { permanent_uuid: "952a9b1a252747a8acdcfa8e04debe30" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34183 } attrs { promote: false } } peers { permanent_uuid: "60abdd24a53146259072b5a6b7220ec0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39371 } attrs { promote: false } } }
I20260929 17:53:30.505252 7788 sys_catalog.cc:458] T 00000000000000000000000000000000 P 952a9b1a252747a8acdcfa8e04debe30 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 17:53:30.506013 7346 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60abdd24a53146259072b5a6b7220ec0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "14cabac4c134442b901728b1c630deb5" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "14cabac4c134442b901728b1c630deb5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39863 } } peers { permanent_uuid: "952a9b1a252747a8acdcfa8e04debe30" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34183 } attrs { promote: false } } peers { permanent_uuid: "60abdd24a53146259072b5a6b7220ec0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39371 } attrs { promote: false } } }
I20260929 17:53:30.506353 7346 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60abdd24a53146259072b5a6b7220ec0 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 17:53:30.513826 7651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:30.507947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43951 (local address 127.0.0.1:49772)
0929 17:53:30.508160 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:30.508174 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 17:53:30.508248 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:53:30.509912 (+ 1664us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:53:30.509915 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:53:30.509934 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:53:30.510038 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:30.510045 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:30.511152 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:30.511155 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:30.512192 (+ 1037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:30.512201 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:30.513386 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:30.513389 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:30.513470 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:30.513493 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 17:53:30.513557 (+ 64us) client_negotiation.cc:240] Negotiation successful
0929 17:53:30.513685 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"mutex_wait_us":31}
I20260929 17:53:30.514031 7789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:30.508197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46193 (local address 127.0.0.1:43320)
0929 17:53:30.508478 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:30.508516 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 17:53:30.508623 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:53:30.510264 (+ 1641us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:53:30.510267 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:53:30.510285 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:53:30.510365 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:30.510371 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:30.511352 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:30.511355 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:30.512516 (+ 1161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:30.512525 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:30.513588 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:30.513592 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:30.513748 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:30.513773 (+ 25us) client_negotiation.cc:769] Sending connection context
0929 17:53:30.513843 (+ 70us) client_negotiation.cc:240] Negotiation successful
0929 17:53:30.513901 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":222,"mutex_wait_us":56,"thread_start_us":112,"threads_started":1}
I20260929 17:53:30.514348 7790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:30.508427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49772 (local address 127.0.98.2:43951)
0929 17:53:30.509407 (+ 980us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:30.509419 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:30.509442 (+ 23us) server_negotiation.cc:407] Connection header received
0929 17:53:30.509523 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:30.509529 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:30.509783 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:30.510098 (+ 315us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:53:30.510244 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:30.511048 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:30.512331 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:30.513059 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:30.513164 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:30.513323 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0929 17:53:30.514089 (+ 766us) server_negotiation.cc:299] Negotiation successful
0929 17:53:30.514196 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":868,"thread_start_us":106,"threads_started":1}
I20260929 17:53:30.514405 7792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:30.508892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43320 (local address 127.0.98.1:46193)
0929 17:53:30.509713 (+ 821us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:30.509724 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:30.509744 (+ 20us) server_negotiation.cc:407] Connection header received
0929 17:53:30.509812 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:30.509817 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:30.509996 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:30.510131 (+ 135us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:53:30.510474 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:30.511241 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:30.512632 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:30.513455 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:30.513572 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:30.513709 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0929 17:53:30.514075 (+ 366us) server_negotiation.cc:299] Negotiation successful
0929 17:53:30.514285 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":729,"thread_start_us":119,"threads_started":1}
I20260929 17:53:30.515269 7791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:30.508496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41685 (local address 127.0.0.1:56238)
0929 17:53:30.509071 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:30.509083 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 17:53:30.509158 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:53:30.511715 (+ 2557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:53:30.511718 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:53:30.511736 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:53:30.511823 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:30.511828 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:30.513055 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:30.513059 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:30.513980 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:30.513988 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:30.514936 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:30.514940 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:30.515025 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:30.515049 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 17:53:30.515093 (+ 44us) client_negotiation.cc:240] Negotiation successful
0929 17:53:30.515152 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":487,"mutex_wait_us":50,"thread_start_us":79,"threads_started":1}
I20260929 17:53:30.515838 7793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:30.509306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56238 (local address 127.0.98.3:41685)
0929 17:53:30.511275 (+ 1969us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:30.511284 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:30.511303 (+ 19us) server_negotiation.cc:407] Connection header received
0929 17:53:30.511372 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:30.511377 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:30.511563 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:30.511967 (+ 404us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:53:30.512088 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:30.512945 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:30.514100 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:30.514809 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:30.515495 (+ 686us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:30.515628 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0929 17:53:30.515666 (+ 38us) server_negotiation.cc:299] Negotiation successful
0929 17:53:30.515735 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1885,"thread_start_us":69,"threads_started":1}
I20260929 17:53:30.517482 7557 tablet_service.cc:1467] Processing CreateTablet for tablet ac08bcfd257a4f1dac3411c804a0a76a (DEFAULT_TABLE table=test-table [id=8309ba97e3c04eb9be176050e5b0db20]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 17:53:30.518399 7557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac08bcfd257a4f1dac3411c804a0a76a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:53:30.518118 7695 tablet_service.cc:1467] Processing CreateTablet for tablet ac08bcfd257a4f1dac3411c804a0a76a (DEFAULT_TABLE table=test-table [id=8309ba97e3c04eb9be176050e5b0db20]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 17:53:30.518909 7695 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac08bcfd257a4f1dac3411c804a0a76a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:53:30.525336 7794 tablet_bootstrap.cc:492] T ac08bcfd257a4f1dac3411c804a0a76a P c4d0d8b2d8994344a6ec327d5c389bba: Bootstrap starting.
I20260929 17:53:30.526364 7795 tablet_bootstrap.cc:492] T ac08bcfd257a4f1dac3411c804a0a76a P ade19fb6904045318b25a43744e7c3f3: Bootstrap starting.
I20260929 17:53:30.526211 7420 tablet_service.cc:1467] Processing CreateTablet for tablet ac08bcfd257a4f1dac3411c804a0a76a (DEFAULT_TABLE table=test-table [id=8309ba97e3c04eb9be176050e5b0db20]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 17:53:30.527240 7420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac08bcfd257a4f1dac3411c804a0a76a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:53:30.527796 7794 tablet_bootstrap.cc:654] T ac08bcfd257a4f1dac3411c804a0a76a P c4d0d8b2d8994344a6ec327d5c389bba: Neither blocks nor log segments found. Creating new log.
I20260929 17:53:30.528355 7795 tablet_bootstrap.cc:654] T ac08bcfd257a4f1dac3411c804a0a76a P ade19fb6904045318b25a43744e7c3f3: Neither blocks nor log segments found. Creating new log.
I20260929 17:53:30.528638 7794 log.cc:824] T ac08bcfd257a4f1dac3411c804a0a76a P c4d0d8b2d8994344a6ec327d5c389bba: Log is configured to *not* fsync() on all Append() calls
I20260929 17:53:30.529136 7795 log.cc:824] T ac08bcfd257a4f1dac3411c804a0a76a P ade19fb6904045318b25a43744e7c3f3: Log is configured to *not* fsync() on all Append() calls
I20260929 17:53:30.530848 7794 tablet_bootstrap.cc:492] T ac08bcfd257a4f1dac3411c804a0a76a P c4d0d8b2d8994344a6ec327d5c389bba: No bootstrap required, opened a new log
I20260929 17:53:30.530848 7795 tablet_bootstrap.cc:492] T ac08bcfd257a4f1dac3411c804a0a76a P ade19fb6904045318b25a43744e7c3f3: No bootstrap required, opened a new log
I20260929 17:53:30.531026 7794 ts_tablet_manager.cc:1397] T ac08bcfd257a4f1dac3411c804a0a76a P c4d0d8b2d8994344a6ec327d5c389bba: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 17:53:30.531026 7795 ts_tablet_manager.cc:1397] T ac08bcfd257a4f1dac3411c804a0a76a P ade19fb6904045318b25a43744e7c3f3: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260929 17:53:30.534219 7798 tablet_bootstrap.cc:492] T ac08bcfd257a4f1dac3411c804a0a76a P 7cb83ff6a04043169a3cf2e232462596: Bootstrap starting.
I20260929 17:53:30.535203 7794 raft_consensus.cc:348] T ac08bcfd257a4f1dac3411c804a0a76a P c4d0d8b2d8994344a6ec327d5c389bba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cb83ff6a04043169a3cf2e232462596" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46193 } } peers { permanent_uuid: "ade19fb6904045318b25a43744e7c3f3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43951 } } peers { permanent_uuid: "c4d0d8b2d8994344a6ec327d5c389bba" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41685 } }
I20260929 17:53:30.535290 7795 raft_consensus.cc:348] T ac08bcfd257a4f1dac3411c804a0a76a P ade19fb6904045318b25a43744e7c3f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cb83ff6a04043169a3cf2e232462596" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46193 } } peers { permanent_uuid: "ade19fb6904045318b25a43744e7c3f3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43951 } } peers { permanent_uuid: "c4d0d8b2d8994344a6ec327d5c389bba" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41685 } }
I20260929 17:53:30.535478 7795 raft_consensus.cc:374] T ac08bcfd257a4f1dac3411c804a0a76a P ade19fb6904045318b25a43744e7c3f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:53:30.535480 7794 raft_consensus.cc:374] T ac08bcfd257a4f1dac3411c804a0a76a P c4d0d8b2d8994344a6ec327d5c389bba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:53:30.535531 7794 raft_consensus.cc:729] T ac08bcfd257a4f1dac3411c804a0a76a P c4d0d8b2d8994344a6ec327d5c389bba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4d0d8b2d8994344a6ec327d5c389bba, State: Initialized, Role: FOLLOWER
I20260929 17:53:30.535538 7795 raft_consensus.cc:729] T ac08bcfd257a4f1dac3411c804a0a76a P ade19fb6904045318b25a43744e7c3f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ade19fb6904045318b25a43744e7c3f3, State: Initialized, Role: FOLLOWER
I20260929 17:53:30.535992 7794 consensus_queue.cc:260] T ac08bcfd257a4f1dac3411c804a0a76a P c4d0d8b2d8994344a6ec327d5c389bba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cb83ff6a04043169a3cf2e232462596" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46193 } } peers { permanent_uuid: "ade19fb6904045318b25a43744e7c3f3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43951 } } peers { permanent_uuid: "c4d0d8b2d8994344a6ec327d5c389bba" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41685 } }
I20260929 17:53:30.535992 7795 consensus_queue.cc:260] T ac08bcfd257a4f1dac3411c804a0a76a P ade19fb6904045318b25a43744e7c3f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cb83ff6a04043169a3cf2e232462596" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46193 } } peers { permanent_uuid: "ade19fb6904045318b25a43744e7c3f3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43951 } } peers { permanent_uuid: "c4d0d8b2d8994344a6ec327d5c389bba" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41685 } }
I20260929 17:53:30.536963 7795 ts_tablet_manager.cc:1428] T ac08bcfd257a4f1dac3411c804a0a76a P ade19fb6904045318b25a43744e7c3f3: Time spent starting tablet: real 0.006s user 0.005s sys 0.002s
I20260929 17:53:30.536964 7794 ts_tablet_manager.cc:1428] T ac08bcfd257a4f1dac3411c804a0a76a P c4d0d8b2d8994344a6ec327d5c389bba: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260929 17:53:30.537376 7624 heartbeater.cc:503] Master 127.0.98.62:39863 was elected leader, sending a full tablet report...
I20260929 17:53:30.537426 7762 heartbeater.cc:503] Master 127.0.98.62:39863 was elected leader, sending a full tablet report...
I20260929 17:53:30.540308 7798 tablet_bootstrap.cc:654] T ac08bcfd257a4f1dac3411c804a0a76a P 7cb83ff6a04043169a3cf2e232462596: Neither blocks nor log segments found. Creating new log.
I20260929 17:53:30.541633 7798 log.cc:824] T ac08bcfd257a4f1dac3411c804a0a76a P 7cb83ff6a04043169a3cf2e232462596: Log is configured to *not* fsync() on all Append() calls
I20260929 17:53:30.543686 7798 tablet_bootstrap.cc:492] T ac08bcfd257a4f1dac3411c804a0a76a P 7cb83ff6a04043169a3cf2e232462596: No bootstrap required, opened a new log
I20260929 17:53:30.543898 7798 ts_tablet_manager.cc:1397] T ac08bcfd257a4f1dac3411c804a0a76a P 7cb83ff6a04043169a3cf2e232462596: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.000s
I20260929 17:53:30.547461 7798 raft_consensus.cc:348] T ac08bcfd257a4f1dac3411c804a0a76a P 7cb83ff6a04043169a3cf2e232462596 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cb83ff6a04043169a3cf2e232462596" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46193 } } peers { permanent_uuid: "ade19fb6904045318b25a43744e7c3f3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43951 } } peers { permanent_uuid: "c4d0d8b2d8994344a6ec327d5c389bba" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41685 } }
I20260929 17:53:30.547683 7798 raft_consensus.cc:374] T ac08bcfd257a4f1dac3411c804a0a76a P 7cb83ff6a04043169a3cf2e232462596 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:53:30.547737 7798 raft_consensus.cc:729] T ac08bcfd257a4f1dac3411c804a0a76a P 7cb83ff6a04043169a3cf2e232462596 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7cb83ff6a04043169a3cf2e232462596, State: Initialized, Role: FOLLOWER
I20260929 17:53:30.548159 7798 consensus_queue.cc:260] T ac08bcfd257a4f1dac3411c804a0a76a P 7cb83ff6a04043169a3cf2e232462596 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cb83ff6a04043169a3cf2e232462596" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46193 } } peers { permanent_uuid: "ade19fb6904045318b25a43744e7c3f3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43951 } } peers { permanent_uuid: "c4d0d8b2d8994344a6ec327d5c389bba" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41685 } }
I20260929 17:53:30.548806 7487 heartbeater.cc:503] Master 127.0.98.62:39863 was elected leader, sending a full tablet report...
I20260929 17:53:30.549146 7798 ts_tablet_manager.cc:1428] T ac08bcfd257a4f1dac3411c804a0a76a P 7cb83ff6a04043169a3cf2e232462596: Time spent starting tablet: real 0.005s user 0.007s sys 0.000s
W20260929 17:53:30.561285 7489 tablet.cc:2369] T ac08bcfd257a4f1dac3411c804a0a76a P 7cb83ff6a04043169a3cf2e232462596: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 17:53:30.661566 7802 raft_consensus.cc:482] T ac08bcfd257a4f1dac3411c804a0a76a P 7cb83ff6a04043169a3cf2e232462596 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 17:53:30.661769 7802 raft_consensus.cc:504] T ac08bcfd257a4f1dac3411c804a0a76a P 7cb83ff6a04043169a3cf2e232462596 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cb83ff6a04043169a3cf2e232462596" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46193 } } peers { permanent_uuid: "ade19fb6904045318b25a43744e7c3f3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43951 } } peers { permanent_uuid: "c4d0d8b2d8994344a6ec327d5c389bba" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41685 } }
I20260929 17:53:30.662827 7802 leader_election.cc:288] T ac08bcfd257a4f1dac3411c804a0a76a P 7cb83ff6a04043169a3cf2e232462596 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ade19fb6904045318b25a43744e7c3f3 (127.0.98.2:43951), c4d0d8b2d8994344a6ec327d5c389bba (127.0.98.3:41685)
I20260929 17:53:30.666880 7790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:30.663131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43303 (local address 127.0.98.2:43951)
0929 17:53:30.663290 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:30.663295 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:30.663417 (+ 122us) server_negotiation.cc:407] Connection header received
0929 17:53:30.663564 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:30.663566 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:30.663609 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:30.663669 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:53:30.663966 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:30.664770 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:30.665920 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:30.666330 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:30.666424 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:30.666485 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0929 17:53:30.666683 (+ 198us) server_negotiation.cc:299] Negotiation successful
0929 17:53:30.666757 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260929 17:53:30.666901 7793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:30.663155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47883 (local address 127.0.98.3:41685)
0929 17:53:30.663340 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:30.663345 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:30.663419 (+ 74us) server_negotiation.cc:407] Connection header received
0929 17:53:30.663564 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:30.663566 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:30.663610 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:30.663673 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:53:30.664016 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:30.664769 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:30.665926 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:30.666316 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:30.666408 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:30.666473 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0929 17:53:30.666708 (+ 235us) server_negotiation.cc:299] Negotiation successful
0929 17:53:30.666789 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114}
I20260929 17:53:30.666913 7804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:30.663031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43951 (local address 127.0.98.1:43303)
0929 17:53:30.663328 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:30.663342 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 17:53:30.663466 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:53:30.663691 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:53:30.663693 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:53:30.663708 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:53:30.663855 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:30.663861 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:30.664919 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:30.664922 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:30.665794 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:30.665803 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:30.666441 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:30.666445 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:30.666564 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:30.666592 (+ 28us) client_negotiation.cc:769] Sending connection context
0929 17:53:30.666655 (+ 63us) client_negotiation.cc:240] Negotiation successful
0929 17:53:30.666744 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":214,"mutex_wait_us":58,"spinlock_wait_cycles":8960,"thread_start_us":105,"threads_started":1}
I20260929 17:53:30.666919 7803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:30.663057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41685 (local address 127.0.98.1:47883)
0929 17:53:30.663338 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:30.663350 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 17:53:30.663465 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:53:30.663702 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:53:30.663705 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:53:30.663719 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:53:30.663866 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:30.663873 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:30.664890 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:30.664892 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:30.665793 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:30.665802 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:30.666434 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:30.666438 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:30.666574 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:30.666602 (+ 28us) client_negotiation.cc:769] Sending connection context
0929 17:53:30.666666 (+ 64us) client_negotiation.cc:240] Negotiation successful
0929 17:53:30.666764 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":226,"mutex_wait_us":129,"spinlock_wait_cycles":7296,"thread_start_us":66,"threads_started":1}
I20260929 17:53:30.667481 7577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac08bcfd257a4f1dac3411c804a0a76a" candidate_uuid: "7cb83ff6a04043169a3cf2e232462596" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ade19fb6904045318b25a43744e7c3f3" is_pre_election: true
I20260929 17:53:30.667665 7715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac08bcfd257a4f1dac3411c804a0a76a" candidate_uuid: "7cb83ff6a04043169a3cf2e232462596" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4d0d8b2d8994344a6ec327d5c389bba" is_pre_election: true
I20260929 17:53:30.667773 7577 raft_consensus.cc:2454] T ac08bcfd257a4f1dac3411c804a0a76a P ade19fb6904045318b25a43744e7c3f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7cb83ff6a04043169a3cf2e232462596 in term 0.
I20260929 17:53:30.667933 7715 raft_consensus.cc:2454] T ac08bcfd257a4f1dac3411c804a0a76a P c4d0d8b2d8994344a6ec327d5c389bba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7cb83ff6a04043169a3cf2e232462596 in term 0.
I20260929 17:53:30.668273 7376 leader_election.cc:302] T ac08bcfd257a4f1dac3411c804a0a76a P 7cb83ff6a04043169a3cf2e232462596 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 7cb83ff6a04043169a3cf2e232462596, ade19fb6904045318b25a43744e7c3f3, c4d0d8b2d8994344a6ec327d5c389bba; no voters:
I20260929 17:53:30.668542 7802 raft_consensus.cc:2781] T ac08bcfd257a4f1dac3411c804a0a76a P 7cb83ff6a04043169a3cf2e232462596 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 17:53:30.668615 7802 raft_consensus.cc:482] T ac08bcfd257a4f1dac3411c804a0a76a P 7cb83ff6a04043169a3cf2e232462596 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 17:53:30.668646 7802 raft_consensus.cc:3037] T ac08bcfd257a4f1dac3411c804a0a76a P 7cb83ff6a04043169a3cf2e232462596 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:53:30.669628 7802 raft_consensus.cc:504] T ac08bcfd257a4f1dac3411c804a0a76a P 7cb83ff6a04043169a3cf2e232462596 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cb83ff6a04043169a3cf2e232462596" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46193 } } peers { permanent_uuid: "ade19fb6904045318b25a43744e7c3f3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43951 } } peers { permanent_uuid: "c4d0d8b2d8994344a6ec327d5c389bba" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41685 } }
I20260929 17:53:30.669978 7802 leader_election.cc:288] T ac08bcfd257a4f1dac3411c804a0a76a P 7cb83ff6a04043169a3cf2e232462596 [CANDIDATE]: Term 1 election: Requested vote from peers ade19fb6904045318b25a43744e7c3f3 (127.0.98.2:43951), c4d0d8b2d8994344a6ec327d5c389bba (127.0.98.3:41685)
I20260929 17:53:30.670343 7577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac08bcfd257a4f1dac3411c804a0a76a" candidate_uuid: "7cb83ff6a04043169a3cf2e232462596" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ade19fb6904045318b25a43744e7c3f3"
I20260929 17:53:30.670367 7715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac08bcfd257a4f1dac3411c804a0a76a" candidate_uuid: "7cb83ff6a04043169a3cf2e232462596" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4d0d8b2d8994344a6ec327d5c389bba"
I20260929 17:53:30.670470 7577 raft_consensus.cc:3037] T ac08bcfd257a4f1dac3411c804a0a76a P ade19fb6904045318b25a43744e7c3f3 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:53:30.670470 7715 raft_consensus.cc:3037] T ac08bcfd257a4f1dac3411c804a0a76a P c4d0d8b2d8994344a6ec327d5c389bba [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:53:30.671720 7577 raft_consensus.cc:2454] T ac08bcfd257a4f1dac3411c804a0a76a P ade19fb6904045318b25a43744e7c3f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7cb83ff6a04043169a3cf2e232462596 in term 1.
I20260929 17:53:30.672005 7376 leader_election.cc:302] T ac08bcfd257a4f1dac3411c804a0a76a P 7cb83ff6a04043169a3cf2e232462596 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7cb83ff6a04043169a3cf2e232462596, ade19fb6904045318b25a43744e7c3f3; no voters:
I20260929 17:53:30.672168 7802 raft_consensus.cc:2781] T ac08bcfd257a4f1dac3411c804a0a76a P 7cb83ff6a04043169a3cf2e232462596 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 17:53:30.672212 7715 raft_consensus.cc:2454] T ac08bcfd257a4f1dac3411c804a0a76a P c4d0d8b2d8994344a6ec327d5c389bba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7cb83ff6a04043169a3cf2e232462596 in term 1.
I20260929 17:53:30.672336 7802 raft_consensus.cc:686] T ac08bcfd257a4f1dac3411c804a0a76a P 7cb83ff6a04043169a3cf2e232462596 [term 1 LEADER]: Becoming Leader. State: Replica: 7cb83ff6a04043169a3cf2e232462596, State: Running, Role: LEADER
I20260929 17:53:30.672899 7802 consensus_queue.cc:237] T ac08bcfd257a4f1dac3411c804a0a76a P 7cb83ff6a04043169a3cf2e232462596 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cb83ff6a04043169a3cf2e232462596" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46193 } } peers { permanent_uuid: "ade19fb6904045318b25a43744e7c3f3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43951 } } peers { permanent_uuid: "c4d0d8b2d8994344a6ec327d5c389bba" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41685 } }
I20260929 17:53:30.676209 6523 catalog_manager.cc:5495] T ac08bcfd257a4f1dac3411c804a0a76a P 7cb83ff6a04043169a3cf2e232462596 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7cb83ff6a04043169a3cf2e232462596 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "7cb83ff6a04043169a3cf2e232462596" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cb83ff6a04043169a3cf2e232462596" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ade19fb6904045318b25a43744e7c3f3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43951 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c4d0d8b2d8994344a6ec327d5c389bba" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41685 } health_report { overall_health: UNKNOWN } } }
I20260929 17:53:30.700680 7792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:30.697576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43330 (local address 127.0.98.1:46193)
0929 17:53:30.697685 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:30.697688 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:30.697700 (+ 12us) server_negotiation.cc:407] Connection header received
0929 17:53:30.697806 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:30.697808 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:30.697861 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:30.697946 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 17:53:30.698138 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:30.698640 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:30.699423 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:30.699620 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:30.699689 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:30.699939 (+ 250us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 17:53:30.700352 (+ 413us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 17:53:30.700438 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0929 17:53:30.700495 (+ 57us) server_negotiation.cc:299] Negotiation successful
0929 17:53:30.700545 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260929 17:53:30.706291 7715 raft_consensus.cc:1261] T ac08bcfd257a4f1dac3411c804a0a76a P c4d0d8b2d8994344a6ec327d5c389bba [term 1 FOLLOWER]: Refusing update from remote peer 7cb83ff6a04043169a3cf2e232462596: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 17:53:30.706290 7577 raft_consensus.cc:1261] T ac08bcfd257a4f1dac3411c804a0a76a P ade19fb6904045318b25a43744e7c3f3 [term 1 FOLLOWER]: Refusing update from remote peer 7cb83ff6a04043169a3cf2e232462596: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 17:53:30.706931 7805 consensus_queue.cc:1035] T ac08bcfd257a4f1dac3411c804a0a76a P 7cb83ff6a04043169a3cf2e232462596 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ade19fb6904045318b25a43744e7c3f3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 17:53:30.707113 7802 consensus_queue.cc:1035] T ac08bcfd257a4f1dac3411c804a0a76a P 7cb83ff6a04043169a3cf2e232462596 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4d0d8b2d8994344a6ec327d5c389bba" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41685 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 17:53:30.725823 6521 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40090:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 17:53:30.725976 6521 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40090:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 17:53:30.729588 6521 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 14cabac4c134442b901728b1c630deb5: Sending DeleteTablet for 3 replicas of tablet ac08bcfd257a4f1dac3411c804a0a76a
I20260929 17:53:30.730638 7695 tablet_service.cc:1514] Processing DeleteTablet for tablet ac08bcfd257a4f1dac3411c804a0a76a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 17:53:30 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56238
I20260929 17:53:30.730633 7420 tablet_service.cc:1514] Processing DeleteTablet for tablet ac08bcfd257a4f1dac3411c804a0a76a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 17:53:30 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43320
I20260929 17:53:30.731384 7819 tablet_replica.cc:330] T ac08bcfd257a4f1dac3411c804a0a76a P 7cb83ff6a04043169a3cf2e232462596: stopping tablet replica
I20260929 17:53:30.731707 7819 raft_consensus.cc:2229] T ac08bcfd257a4f1dac3411c804a0a76a P 7cb83ff6a04043169a3cf2e232462596 [term 1 LEADER]: Raft consensus shutting down.
I20260929 17:53:30.731678 7557 tablet_service.cc:1514] Processing DeleteTablet for tablet ac08bcfd257a4f1dac3411c804a0a76a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 17:53:30 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49772
I20260929 17:53:30.732091 7819 raft_consensus.cc:2258] T ac08bcfd257a4f1dac3411c804a0a76a P 7cb83ff6a04043169a3cf2e232462596 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 17:53:30.732847 7821 tablet_replica.cc:330] T ac08bcfd257a4f1dac3411c804a0a76a P ade19fb6904045318b25a43744e7c3f3: stopping tablet replica
I20260929 17:53:30.733230 7821 raft_consensus.cc:2229] T ac08bcfd257a4f1dac3411c804a0a76a P ade19fb6904045318b25a43744e7c3f3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 17:53:30.733597 7821 raft_consensus.cc:2258] T ac08bcfd257a4f1dac3411c804a0a76a P ade19fb6904045318b25a43744e7c3f3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 17:53:30.733666 7819 ts_tablet_manager.cc:1905] T ac08bcfd257a4f1dac3411c804a0a76a P 7cb83ff6a04043169a3cf2e232462596: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 17:53:30.733767 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskup3Soe/build/debug/bin/kudu with pid 6491
I20260929 17:53:30.735924 7821 ts_tablet_manager.cc:1905] T ac08bcfd257a4f1dac3411c804a0a76a P ade19fb6904045318b25a43744e7c3f3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 17:53:30.736455 7819 ts_tablet_manager.cc:1918] T ac08bcfd257a4f1dac3411c804a0a76a P 7cb83ff6a04043169a3cf2e232462596: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 17:53:30.736543 7819 log.cc:1196] T ac08bcfd257a4f1dac3411c804a0a76a P 7cb83ff6a04043169a3cf2e232462596: Deleting WAL directory at /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704375975685-392-0/minicluster-data/ts-0/wal/wals/ac08bcfd257a4f1dac3411c804a0a76a
I20260929 17:53:30.736821 7819 ts_tablet_manager.cc:1939] T ac08bcfd257a4f1dac3411c804a0a76a P 7cb83ff6a04043169a3cf2e232462596: Deleting consensus metadata
I20260929 17:53:30.737994 7818 tablet_replica.cc:330] T ac08bcfd257a4f1dac3411c804a0a76a P c4d0d8b2d8994344a6ec327d5c389bba: stopping tablet replica
I20260929 17:53:30.738312 7818 raft_consensus.cc:2229] T ac08bcfd257a4f1dac3411c804a0a76a P c4d0d8b2d8994344a6ec327d5c389bba [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 17:53:30.738662 7818 raft_consensus.cc:2258] T ac08bcfd257a4f1dac3411c804a0a76a P c4d0d8b2d8994344a6ec327d5c389bba [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 17:53:30.738847 7821 ts_tablet_manager.cc:1918] T ac08bcfd257a4f1dac3411c804a0a76a P ade19fb6904045318b25a43744e7c3f3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 17:53:30.738973 7821 log.cc:1196] T ac08bcfd257a4f1dac3411c804a0a76a P ade19fb6904045318b25a43744e7c3f3: Deleting WAL directory at /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704375975685-392-0/minicluster-data/ts-1/wal/wals/ac08bcfd257a4f1dac3411c804a0a76a
I20260929 17:53:30.739719 7821 ts_tablet_manager.cc:1939] T ac08bcfd257a4f1dac3411c804a0a76a P ade19fb6904045318b25a43744e7c3f3: Deleting consensus metadata
I20260929 17:53:30.740248 7818 ts_tablet_manager.cc:1905] T ac08bcfd257a4f1dac3411c804a0a76a P c4d0d8b2d8994344a6ec327d5c389bba: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260929 17:53:30.741747 7373 connection.cc:535] server connection from 127.0.0.1:43320 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260929 17:53:30.742141 7512 connection.cc:535] server connection from 127.0.0.1:49772 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260929 17:53:30.743280 7818 ts_tablet_manager.cc:1918] T ac08bcfd257a4f1dac3411c804a0a76a P c4d0d8b2d8994344a6ec327d5c389bba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 17:53:30.743379 7818 log.cc:1196] T ac08bcfd257a4f1dac3411c804a0a76a P c4d0d8b2d8994344a6ec327d5c389bba: Deleting WAL directory at /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704375975685-392-0/minicluster-data/ts-2/wal/wals/ac08bcfd257a4f1dac3411c804a0a76a
I20260929 17:53:30.743682 7818 ts_tablet_manager.cc:1939] T ac08bcfd257a4f1dac3411c804a0a76a P c4d0d8b2d8994344a6ec327d5c389bba: Deleting consensus metadata
W20260929 17:53:30.744913 7647 connection.cc:441] server connection from 127.0.0.1:56238 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56238 (request call id 1) could send its response
I20260929 17:53:30.754467 7245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:30.732266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41516 (local address 127.0.98.61:34183)
0929 17:53:30.732379 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:30.732383 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:30.743364 (+ 10981us) server_negotiation.cc:407] Connection header received
0929 17:53:30.743587 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:30.743591 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:30.743647 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:30.743701 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:53:30.744783 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:30.745448 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:30.746412 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:30.746707 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:30.746779 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:30.749870 (+ 3091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:53:30.749874 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:53:30.749876 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:53:30.749903 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:53:30.751863 (+ 1960us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:53:30.752318 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:53:30.752321 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:53:30.752323 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:53:30.752404 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:53:30.752696 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:53:30.752700 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:53:30.752702 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:53:30.752910 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:53:30.754130 (+ 1220us) server_negotiation.cc:1035] Waiting for connection context
0929 17:53:30.754193 (+ 63us) server_negotiation.cc:299] Negotiation successful
0929 17:53:30.754298 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 17:53:30.755307 7492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:30.745842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46858 (local address 127.0.98.60:39371)
0929 17:53:30.745952 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:30.745957 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:30.746310 (+ 353us) server_negotiation.cc:407] Connection header received
0929 17:53:30.746479 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:30.746482 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:30.746528 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:30.746574 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:53:30.748288 (+ 1714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:30.749050 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:30.750183 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:30.750507 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:30.750582 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:30.752110 (+ 1528us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:53:30.752114 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:53:30.752117 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:53:30.752146 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:53:30.753617 (+ 1471us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:53:30.754035 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:53:30.754038 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:53:30.754040 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:53:30.754115 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:53:30.754398 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:53:30.754402 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:53:30.754403 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:53:30.754611 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:53:30.754711 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0929 17:53:30.755047 (+ 336us) server_negotiation.cc:299] Negotiation successful
0929 17:53:30.755155 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260929 17:53:30.757628 7776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:30.746472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41528 (local address 127.0.98.61:34183)
0929 17:53:30.746717 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:30.746720 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:30.746977 (+ 257us) server_negotiation.cc:407] Connection header received
0929 17:53:30.747133 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:30.747136 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:30.747178 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:30.747224 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:53:30.748083 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:30.748865 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:30.750093 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:30.750440 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:30.750510 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:30.751999 (+ 1489us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:53:30.752003 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:53:30.752006 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:53:30.752030 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:53:30.754014 (+ 1984us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:53:30.756344 (+ 2330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:53:30.756348 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:53:30.756350 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:53:30.756427 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:53:30.756705 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:53:30.756708 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:53:30.756710 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:53:30.756901 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:53:30.756991 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0929 17:53:30.757360 (+ 369us) server_negotiation.cc:299] Negotiation successful
0929 17:53:30.757492 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170}
I20260929 17:53:30.761533 7772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:30.732121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46846 (local address 127.0.98.60:39371)
0929 17:53:30.732227 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:30.732232 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:30.753822 (+ 21590us) server_negotiation.cc:407] Connection header received
0929 17:53:30.753980 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:30.753983 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:30.754034 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:30.754081 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:53:30.754784 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:30.755501 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:30.756426 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:30.756725 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:30.756796 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:30.758664 (+ 1868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:53:30.758667 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:53:30.758670 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:53:30.758692 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:53:30.760017 (+ 1325us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:53:30.760374 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:53:30.760377 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:53:30.760379 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:53:30.760452 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:53:30.760679 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:53:30.760682 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:53:30.760683 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:53:30.760848 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:53:30.760954 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0929 17:53:30.761268 (+ 314us) server_negotiation.cc:299] Negotiation successful
0929 17:53:30.761385 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260929 17:53:31.719904 7828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:31.719513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39863 (local address 127.0.98.2:50473)
0929 17:53:31.719717 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:31.719795 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":67,"threads_started":1}
W20260929 17:53:31.720214 7624 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:39863 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:39863: connect: Connection refused (error 111)
I20260929 17:53:31.720881 7829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:31.720505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39863 (local address 127.0.98.1:46137)
0929 17:53:31.720729 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:31.720796 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":57,"threads_started":1}
I20260929 17:53:31.721150 7830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:31.720567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39863 (local address 127.0.98.3:55483)
0929 17:53:31.720957 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:31.721061 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":63,"threads_started":1}
W20260929 17:53:31.721269 7487 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:39863 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:39863: connect: Connection refused (error 111)
W20260929 17:53:31.721385 7762 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:39863 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:39863: connect: Connection refused (error 111)
I20260929 17:53:32.230640 7831 raft_consensus.cc:482] T 00000000000000000000000000000000 P 952a9b1a252747a8acdcfa8e04debe30 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 14cabac4c134442b901728b1c630deb5)
I20260929 17:53:32.230902 7831 raft_consensus.cc:504] T 00000000000000000000000000000000 P 952a9b1a252747a8acdcfa8e04debe30 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "14cabac4c134442b901728b1c630deb5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39863 } } peers { permanent_uuid: "952a9b1a252747a8acdcfa8e04debe30" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34183 } attrs { promote: false } } peers { permanent_uuid: "60abdd24a53146259072b5a6b7220ec0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39371 } attrs { promote: false } }
I20260929 17:53:32.231096 7832 raft_consensus.cc:482] T 00000000000000000000000000000000 P 60abdd24a53146259072b5a6b7220ec0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 14cabac4c134442b901728b1c630deb5)
I20260929 17:53:32.231254 7832 raft_consensus.cc:504] T 00000000000000000000000000000000 P 60abdd24a53146259072b5a6b7220ec0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "14cabac4c134442b901728b1c630deb5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39863 } } peers { permanent_uuid: "952a9b1a252747a8acdcfa8e04debe30" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34183 } attrs { promote: false } } peers { permanent_uuid: "60abdd24a53146259072b5a6b7220ec0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39371 } attrs { promote: false } }
I20260929 17:53:32.231914 7831 leader_election.cc:288] T 00000000000000000000000000000000 P 952a9b1a252747a8acdcfa8e04debe30 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 14cabac4c134442b901728b1c630deb5 (127.0.98.62:39863), 60abdd24a53146259072b5a6b7220ec0 (127.0.98.60:39371)
I20260929 17:53:32.232105 7832 leader_election.cc:288] T 00000000000000000000000000000000 P 60abdd24a53146259072b5a6b7220ec0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 14cabac4c134442b901728b1c630deb5 (127.0.98.62:39863), 952a9b1a252747a8acdcfa8e04debe30 (127.0.98.61:34183)
I20260929 17:53:32.232577 7834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:32.232126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39863 (local address 127.0.0.1:40552)
0929 17:53:32.232400 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:32.232476 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":75,"threads_started":1}
W20260929 17:53:32.232846 7109 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39863: connect: Connection refused (error 111)
I20260929 17:53:32.232964 7835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:32.232476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39863 (local address 127.0.0.1:40558)
0929 17:53:32.232822 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:32.232885 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":51,"threads_started":1}
W20260929 17:53:32.233117 7293 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39863: connect: Connection refused (error 111)
I20260929 17:53:32.233933 7834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:32.233742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39863 (local address 127.0.0.1:40584)
0929 17:53:32.233847 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:32.233885 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260929 17:53:32.233932 7835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:32.233710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39863 (local address 127.0.0.1:40572)
0929 17:53:32.233831 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:32.233876 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
W20260929 17:53:32.234055 7293 leader_election.cc:334] T 00000000000000000000000000000000 P 60abdd24a53146259072b5a6b7220ec0 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 14cabac4c134442b901728b1c630deb5 (127.0.98.62:39863): Network error: Client connection negotiation failed: client connection to 127.0.98.62:39863: connect: Connection refused (error 111)
W20260929 17:53:32.234105 7109 leader_election.cc:334] T 00000000000000000000000000000000 P 952a9b1a252747a8acdcfa8e04debe30 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 14cabac4c134442b901728b1c630deb5 (127.0.98.62:39863): Network error: Client connection negotiation failed: client connection to 127.0.98.62:39863: connect: Connection refused (error 111)
I20260929 17:53:32.236171 7836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:32.232411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34183 (local address 127.0.0.1:41530)
0929 17:53:32.232733 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:32.232746 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 17:53:32.232833 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:53:32.233148 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:53:32.233151 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:53:32.233166 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:53:32.233275 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:32.233282 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:32.234399 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:32.234402 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:32.235166 (+ 764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:32.235173 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:32.235766 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:32.235770 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:32.235857 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:32.235874 (+ 17us) client_negotiation.cc:769] Sending connection context
0929 17:53:32.235937 (+ 63us) client_negotiation.cc:240] Negotiation successful
0929 17:53:32.236015 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":62,"threads_started":1}
I20260929 17:53:32.236179 7838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:32.232666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41530 (local address 127.0.98.61:34183)
0929 17:53:32.232908 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:32.232913 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:32.232925 (+ 12us) server_negotiation.cc:407] Connection header received
0929 17:53:32.232967 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:32.232970 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:32.233037 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:32.233383 (+ 346us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:53:32.233457 (+ 74us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:32.234273 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:32.235278 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:32.235663 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:32.235751 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:32.235816 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0929 17:53:32.235959 (+ 143us) server_negotiation.cc:299] Negotiation successful
0929 17:53:32.236047 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":188,"thread_start_us":57,"threads_started":1}
I20260929 17:53:32.236208 7833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:32.232135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39371 (local address 127.0.0.1:46868)
0929 17:53:32.232418 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:32.232432 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 17:53:32.232533 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:53:32.233199 (+ 666us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:53:32.233202 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:53:32.233216 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:53:32.233312 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:32.233319 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:32.234385 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:32.234388 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:32.235166 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:32.235174 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:32.235805 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:32.235809 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:32.235893 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:32.235911 (+ 18us) client_negotiation.cc:769] Sending connection context
0929 17:53:32.235972 (+ 61us) client_negotiation.cc:240] Negotiation successful
0929 17:53:32.236069 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":210,"mutex_wait_us":23,"thread_start_us":78,"threads_started":1}
I20260929 17:53:32.236199 7837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:32.232648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46868 (local address 127.0.98.60:39371)
0929 17:53:32.232989 (+ 341us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:32.232993 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:32.233025 (+ 32us) server_negotiation.cc:407] Connection header received
0929 17:53:32.233060 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:32.233063 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:32.233098 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:32.233400 (+ 302us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:53:32.233476 (+ 76us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:32.234253 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:32.235304 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:32.235707 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:32.235795 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:32.235848 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0929 17:53:32.235999 (+ 151us) server_negotiation.cc:299] Negotiation successful
0929 17:53:32.236069 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":292,"thread_start_us":54,"threads_started":1}
I20260929 17:53:32.236581 7133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "60abdd24a53146259072b5a6b7220ec0" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "952a9b1a252747a8acdcfa8e04debe30" is_pre_election: true
I20260929 17:53:32.236792 7133 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 952a9b1a252747a8acdcfa8e04debe30 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60abdd24a53146259072b5a6b7220ec0 in term 1.
I20260929 17:53:32.236846 7308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "952a9b1a252747a8acdcfa8e04debe30" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "60abdd24a53146259072b5a6b7220ec0" is_pre_election: true
I20260929 17:53:32.237051 7308 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 60abdd24a53146259072b5a6b7220ec0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 952a9b1a252747a8acdcfa8e04debe30 in term 1.
I20260929 17:53:32.237131 7291 leader_election.cc:302] T 00000000000000000000000000000000 P 60abdd24a53146259072b5a6b7220ec0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60abdd24a53146259072b5a6b7220ec0, 952a9b1a252747a8acdcfa8e04debe30; no voters: 14cabac4c134442b901728b1c630deb5
I20260929 17:53:32.237353 7832 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 60abdd24a53146259072b5a6b7220ec0 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260929 17:53:32.237396 7108 leader_election.cc:302] T 00000000000000000000000000000000 P 952a9b1a252747a8acdcfa8e04debe30 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60abdd24a53146259072b5a6b7220ec0, 952a9b1a252747a8acdcfa8e04debe30; no voters: 14cabac4c134442b901728b1c630deb5
I20260929 17:53:32.237440 7832 raft_consensus.cc:482] T 00000000000000000000000000000000 P 60abdd24a53146259072b5a6b7220ec0 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 14cabac4c134442b901728b1c630deb5)
I20260929 17:53:32.237490 7832 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 60abdd24a53146259072b5a6b7220ec0 [term 1 FOLLOWER]: Advancing to term 2
I20260929 17:53:32.237691 7831 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 952a9b1a252747a8acdcfa8e04debe30 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260929 17:53:32.237751 7831 raft_consensus.cc:482] T 00000000000000000000000000000000 P 952a9b1a252747a8acdcfa8e04debe30 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 14cabac4c134442b901728b1c630deb5)
I20260929 17:53:32.237794 7831 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 952a9b1a252747a8acdcfa8e04debe30 [term 1 FOLLOWER]: Advancing to term 2
I20260929 17:53:32.238564 7832 raft_consensus.cc:504] T 00000000000000000000000000000000 P 60abdd24a53146259072b5a6b7220ec0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "14cabac4c134442b901728b1c630deb5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39863 } } peers { permanent_uuid: "952a9b1a252747a8acdcfa8e04debe30" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34183 } attrs { promote: false } } peers { permanent_uuid: "60abdd24a53146259072b5a6b7220ec0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39371 } attrs { promote: false } }
I20260929 17:53:32.238809 7831 raft_consensus.cc:504] T 00000000000000000000000000000000 P 952a9b1a252747a8acdcfa8e04debe30 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "14cabac4c134442b901728b1c630deb5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39863 } } peers { permanent_uuid: "952a9b1a252747a8acdcfa8e04debe30" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34183 } attrs { promote: false } } peers { permanent_uuid: "60abdd24a53146259072b5a6b7220ec0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39371 } attrs { promote: false } }
I20260929 17:53:32.238897 7832 leader_election.cc:288] T 00000000000000000000000000000000 P 60abdd24a53146259072b5a6b7220ec0 [CANDIDATE]: Term 2 election: Requested vote from peers 14cabac4c134442b901728b1c630deb5 (127.0.98.62:39863), 952a9b1a252747a8acdcfa8e04debe30 (127.0.98.61:34183)
I20260929 17:53:32.239162 7831 leader_election.cc:288] T 00000000000000000000000000000000 P 952a9b1a252747a8acdcfa8e04debe30 [CANDIDATE]: Term 2 election: Requested vote from peers 14cabac4c134442b901728b1c630deb5 (127.0.98.62:39863), 60abdd24a53146259072b5a6b7220ec0 (127.0.98.60:39371)
I20260929 17:53:32.239369 7836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:32.239016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39863 (local address 127.0.0.1:40588)
0929 17:53:32.239252 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:32.239306 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130}
I20260929 17:53:32.239365 7133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "60abdd24a53146259072b5a6b7220ec0" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "952a9b1a252747a8acdcfa8e04debe30"
I20260929 17:53:32.239413 7308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "952a9b1a252747a8acdcfa8e04debe30" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "60abdd24a53146259072b5a6b7220ec0"
I20260929 17:53:32.239521 7133 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 952a9b1a252747a8acdcfa8e04debe30 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 60abdd24a53146259072b5a6b7220ec0 in current term 2: Already voted for candidate 952a9b1a252747a8acdcfa8e04debe30 in this term.
I20260929 17:53:32.239552 7308 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 60abdd24a53146259072b5a6b7220ec0 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 952a9b1a252747a8acdcfa8e04debe30 in current term 2: Already voted for candidate 60abdd24a53146259072b5a6b7220ec0 in this term.
I20260929 17:53:32.239823 7833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:32.239243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39863 (local address 127.0.0.1:40602)
0929 17:53:32.239713 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:32.239761 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":412}
I20260929 17:53:32.240140 7836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:32.239923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39863 (local address 127.0.0.1:40612)
0929 17:53:32.240040 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:32.240086 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260929 17:53:32.240233 7293 leader_election.cc:334] T 00000000000000000000000000000000 P 60abdd24a53146259072b5a6b7220ec0 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 14cabac4c134442b901728b1c630deb5 (127.0.98.62:39863): Network error: Client connection negotiation failed: client connection to 127.0.98.62:39863: connect: Connection refused (error 111)
I20260929 17:53:32.240299 7293 leader_election.cc:302] T 00000000000000000000000000000000 P 60abdd24a53146259072b5a6b7220ec0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60abdd24a53146259072b5a6b7220ec0; no voters: 14cabac4c134442b901728b1c630deb5, 952a9b1a252747a8acdcfa8e04debe30
I20260929 17:53:32.240432 7832 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 60abdd24a53146259072b5a6b7220ec0 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260929 17:53:32.240655 7833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:32.240201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39863 (local address 127.0.0.1:40628)
0929 17:53:32.240552 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:32.240600 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":288}
W20260929 17:53:32.240751 7109 leader_election.cc:334] T 00000000000000000000000000000000 P 952a9b1a252747a8acdcfa8e04debe30 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 14cabac4c134442b901728b1c630deb5 (127.0.98.62:39863): Network error: Client connection negotiation failed: client connection to 127.0.98.62:39863: connect: Connection refused (error 111)
I20260929 17:53:32.240815 7109 leader_election.cc:302] T 00000000000000000000000000000000 P 952a9b1a252747a8acdcfa8e04debe30 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 952a9b1a252747a8acdcfa8e04debe30; no voters: 14cabac4c134442b901728b1c630deb5, 60abdd24a53146259072b5a6b7220ec0
I20260929 17:53:32.240940 7831 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 952a9b1a252747a8acdcfa8e04debe30 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260929 17:53:32.721139 7841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:32.720719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39863 (local address 127.0.98.2:48453)
0929 17:53:32.720973 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:32.721051 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":90,"threads_started":1}
I20260929 17:53:32.722210 7843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:32.721811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39863 (local address 127.0.98.3:43703)
0929 17:53:32.722064 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:32.722121 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":60,"threads_started":1}
I20260929 17:53:32.722220 7842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:32.721695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39863 (local address 127.0.98.1:40859)
0929 17:53:32.722065 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:32.722124 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":78,"threads_started":1}
I20260929 17:53:33.722082 7844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:33.721699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39863 (local address 127.0.98.2:48925)
0929 17:53:33.721932 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:33.721991 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":72,"threads_started":1}
I20260929 17:53:33.723268 7845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:33.722829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39863 (local address 127.0.98.1:33833)
0929 17:53:33.723074 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:33.723136 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":72,"threads_started":1}
I20260929 17:53:33.723337 7846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:33.722829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39863 (local address 127.0.98.3:47681)
0929 17:53:33.723153 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:33.723205 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":74,"threads_started":1}
I20260929 17:53:34.032866 7847 raft_consensus.cc:482] T 00000000000000000000000000000000 P 952a9b1a252747a8acdcfa8e04debe30 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 17:53:34.032985 7847 raft_consensus.cc:504] T 00000000000000000000000000000000 P 952a9b1a252747a8acdcfa8e04debe30 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "14cabac4c134442b901728b1c630deb5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39863 } } peers { permanent_uuid: "952a9b1a252747a8acdcfa8e04debe30" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34183 } attrs { promote: false } } peers { permanent_uuid: "60abdd24a53146259072b5a6b7220ec0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39371 } attrs { promote: false } }
I20260929 17:53:34.033396 7847 leader_election.cc:288] T 00000000000000000000000000000000 P 952a9b1a252747a8acdcfa8e04debe30 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 14cabac4c134442b901728b1c630deb5 (127.0.98.62:39863), 60abdd24a53146259072b5a6b7220ec0 (127.0.98.60:39371)
I20260929 17:53:34.033879 7308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "952a9b1a252747a8acdcfa8e04debe30" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "60abdd24a53146259072b5a6b7220ec0" is_pre_election: true
I20260929 17:53:34.033981 7848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:34.033565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39863 (local address 127.0.0.1:40894)
0929 17:53:34.033812 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:34.033884 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":95,"threads_started":1}
I20260929 17:53:34.034039 7308 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 60abdd24a53146259072b5a6b7220ec0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 952a9b1a252747a8acdcfa8e04debe30 in term 2.
I20260929 17:53:34.034370 7108 leader_election.cc:302] T 00000000000000000000000000000000 P 952a9b1a252747a8acdcfa8e04debe30 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60abdd24a53146259072b5a6b7220ec0, 952a9b1a252747a8acdcfa8e04debe30; no voters:
I20260929 17:53:34.034538 7847 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 952a9b1a252747a8acdcfa8e04debe30 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260929 17:53:34.034611 7847 raft_consensus.cc:482] T 00000000000000000000000000000000 P 952a9b1a252747a8acdcfa8e04debe30 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 17:53:34.034657 7847 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 952a9b1a252747a8acdcfa8e04debe30 [term 2 FOLLOWER]: Advancing to term 3
I20260929 17:53:34.034775 7848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:34.034536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39863 (local address 127.0.0.1:40910)
0929 17:53:34.034639 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:34.034715 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260929 17:53:34.034893 7109 leader_election.cc:334] T 00000000000000000000000000000000 P 952a9b1a252747a8acdcfa8e04debe30 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 14cabac4c134442b901728b1c630deb5 (127.0.98.62:39863): Network error: Client connection negotiation failed: client connection to 127.0.98.62:39863: connect: Connection refused (error 111)
I20260929 17:53:34.035704 7847 raft_consensus.cc:504] T 00000000000000000000000000000000 P 952a9b1a252747a8acdcfa8e04debe30 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "14cabac4c134442b901728b1c630deb5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39863 } } peers { permanent_uuid: "952a9b1a252747a8acdcfa8e04debe30" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34183 } attrs { promote: false } } peers { permanent_uuid: "60abdd24a53146259072b5a6b7220ec0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39371 } attrs { promote: false } }
I20260929 17:53:34.035995 7847 leader_election.cc:288] T 00000000000000000000000000000000 P 952a9b1a252747a8acdcfa8e04debe30 [CANDIDATE]: Term 3 election: Requested vote from peers 14cabac4c134442b901728b1c630deb5 (127.0.98.62:39863), 60abdd24a53146259072b5a6b7220ec0 (127.0.98.60:39371)
I20260929 17:53:34.036304 7848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:34.036100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39863 (local address 127.0.0.1:40924)
0929 17:53:34.036195 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:34.036244 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260929 17:53:34.036337 7308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "952a9b1a252747a8acdcfa8e04debe30" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "60abdd24a53146259072b5a6b7220ec0"
I20260929 17:53:34.036464 7308 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 60abdd24a53146259072b5a6b7220ec0 [term 2 FOLLOWER]: Advancing to term 3
I20260929 17:53:34.036931 7848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:34.036716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39863 (local address 127.0.0.1:40926)
0929 17:53:34.036802 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:34.036840 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260929 17:53:34.037043 7109 leader_election.cc:334] T 00000000000000000000000000000000 P 952a9b1a252747a8acdcfa8e04debe30 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 14cabac4c134442b901728b1c630deb5 (127.0.98.62:39863): Network error: Client connection negotiation failed: client connection to 127.0.98.62:39863: connect: Connection refused (error 111)
I20260929 17:53:34.037566 7308 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 60abdd24a53146259072b5a6b7220ec0 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 952a9b1a252747a8acdcfa8e04debe30 in term 3.
I20260929 17:53:34.037806 7108 leader_election.cc:302] T 00000000000000000000000000000000 P 952a9b1a252747a8acdcfa8e04debe30 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60abdd24a53146259072b5a6b7220ec0, 952a9b1a252747a8acdcfa8e04debe30; no voters: 14cabac4c134442b901728b1c630deb5
I20260929 17:53:34.037943 7847 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 952a9b1a252747a8acdcfa8e04debe30 [term 3 FOLLOWER]: Leader election won for term 3
I20260929 17:53:34.038100 7847 raft_consensus.cc:686] T 00000000000000000000000000000000 P 952a9b1a252747a8acdcfa8e04debe30 [term 3 LEADER]: Becoming Leader. State: Replica: 952a9b1a252747a8acdcfa8e04debe30, State: Running, Role: LEADER
I20260929 17:53:34.038393 7847 consensus_queue.cc:237] T 00000000000000000000000000000000 P 952a9b1a252747a8acdcfa8e04debe30 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "14cabac4c134442b901728b1c630deb5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39863 } } peers { permanent_uuid: "952a9b1a252747a8acdcfa8e04debe30" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34183 } attrs { promote: false } } peers { permanent_uuid: "60abdd24a53146259072b5a6b7220ec0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39371 } attrs { promote: false } }
I20260929 17:53:34.039371 7850 sys_catalog.cc:455] T 00000000000000000000000000000000 P 952a9b1a252747a8acdcfa8e04debe30 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 952a9b1a252747a8acdcfa8e04debe30. Latest consensus state: current_term: 3 leader_uuid: "952a9b1a252747a8acdcfa8e04debe30" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "14cabac4c134442b901728b1c630deb5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39863 } } peers { permanent_uuid: "952a9b1a252747a8acdcfa8e04debe30" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34183 } attrs { promote: false } } peers { permanent_uuid: "60abdd24a53146259072b5a6b7220ec0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39371 } attrs { promote: false } } }
I20260929 17:53:34.039485 7850 sys_catalog.cc:458] T 00000000000000000000000000000000 P 952a9b1a252747a8acdcfa8e04debe30 [sys.catalog]: This master's current role is: LEADER
I20260929 17:53:34.039781 7852 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 17:53:34.041946 7852 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 17:53:34.042309 7852 catalog_manager.cc:1259] Loaded cluster ID: 206c4021e40644f6bf3327638ec8f5f8
I20260929 17:53:34.042362 7852 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 17:53:34.042949 7852 catalog_manager.cc:1504] Loading token signing keys...
I20260929 17:53:34.043336 7852 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 952a9b1a252747a8acdcfa8e04debe30: Loaded TSK: 0
I20260929 17:53:34.044085 7852 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 17:53:34.117025 7123 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41528:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 17:53:34.120061 7123 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 17:53:34.122938 7848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:34.122698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39863 (local address 127.0.0.1:40944)
0929 17:53:34.122817 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:34.122872 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260929 17:53:34.123049 7308 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 60abdd24a53146259072b5a6b7220ec0 [term 3 FOLLOWER]: Refusing update from remote peer 952a9b1a252747a8acdcfa8e04debe30: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260929 17:53:34.123503 7850 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 952a9b1a252747a8acdcfa8e04debe30 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60abdd24a53146259072b5a6b7220ec0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39371 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260929 17:53:34.123638 7848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:34.123444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39863 (local address 127.0.0.1:40954)
0929 17:53:34.123544 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:34.123586 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260929 17:53:34.123729 7109 consensus_peers.cc:487] T 00000000000000000000000000000000 P 952a9b1a252747a8acdcfa8e04debe30 -> Peer 14cabac4c134442b901728b1c630deb5 (127.0.98.62:39863): Couldn't send request to peer 14cabac4c134442b901728b1c630deb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39863: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260929 17:53:34.124922 7854 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60abdd24a53146259072b5a6b7220ec0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 952a9b1a252747a8acdcfa8e04debe30. Latest consensus state: current_term: 3 leader_uuid: "952a9b1a252747a8acdcfa8e04debe30" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "14cabac4c134442b901728b1c630deb5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39863 } } peers { permanent_uuid: "952a9b1a252747a8acdcfa8e04debe30" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34183 } attrs { promote: false } } peers { permanent_uuid: "60abdd24a53146259072b5a6b7220ec0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39371 } attrs { promote: false } } }
I20260929 17:53:34.125061 7854 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60abdd24a53146259072b5a6b7220ec0 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 17:53:34.126080 7850 sys_catalog.cc:455] T 00000000000000000000000000000000 P 952a9b1a252747a8acdcfa8e04debe30 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "952a9b1a252747a8acdcfa8e04debe30" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "14cabac4c134442b901728b1c630deb5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39863 } } peers { permanent_uuid: "952a9b1a252747a8acdcfa8e04debe30" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34183 } attrs { promote: false } } peers { permanent_uuid: "60abdd24a53146259072b5a6b7220ec0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39371 } attrs { promote: false } } }
I20260929 17:53:34.126197 7850 sys_catalog.cc:458] T 00000000000000000000000000000000 P 952a9b1a252747a8acdcfa8e04debe30 [sys.catalog]: This master's current role is: LEADER
I20260929 17:53:34.126398 7854 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60abdd24a53146259072b5a6b7220ec0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "952a9b1a252747a8acdcfa8e04debe30" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "14cabac4c134442b901728b1c630deb5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39863 } } peers { permanent_uuid: "952a9b1a252747a8acdcfa8e04debe30" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34183 } attrs { promote: false } } peers { permanent_uuid: "60abdd24a53146259072b5a6b7220ec0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39371 } attrs { promote: false } } }
I20260929 17:53:34.126559 7854 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60abdd24a53146259072b5a6b7220ec0 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 17:53:34.135949 7848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:34.132018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41685 (local address 127.0.0.1:56246)
0929 17:53:34.132386 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:34.132401 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 17:53:34.132491 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:53:34.132752 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:53:34.132755 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:53:34.132774 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:53:34.132880 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:34.132887 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:34.133870 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:34.133874 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:34.134659 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:34.134670 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:34.135461 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:34.135465 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:34.135621 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:34.135650 (+ 29us) client_negotiation.cc:769] Sending connection context
0929 17:53:34.135718 (+ 68us) client_negotiation.cc:240] Negotiation successful
0929 17:53:34.135798 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":304,"mutex_wait_us":36}
I20260929 17:53:34.136453 7860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:34.132158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56246 (local address 127.0.98.3:41685)
0929 17:53:34.132424 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:34.132429 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:34.132452 (+ 23us) server_negotiation.cc:407] Connection header received
0929 17:53:34.132591 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:34.132593 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:34.132635 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:34.132733 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:53:34.133038 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:34.133755 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:34.134808 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:34.135351 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:34.135453 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:34.135517 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0929 17:53:34.136226 (+ 709us) server_negotiation.cc:299] Negotiation successful
0929 17:53:34.136311 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":80,"threads_started":1}
I20260929 17:53:34.136889 7861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:34.132693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46193 (local address 127.0.0.1:43346)
0929 17:53:34.133108 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:34.133123 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 17:53:34.133217 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:53:34.133656 (+ 439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:53:34.133659 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:53:34.133674 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:53:34.133778 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:34.133785 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:34.134608 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:34.134612 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:34.135624 (+ 1012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:34.135632 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:34.136512 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:34.136516 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:34.136599 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:34.136620 (+ 21us) client_negotiation.cc:769] Sending connection context
0929 17:53:34.136686 (+ 66us) client_negotiation.cc:240] Negotiation successful
0929 17:53:34.136752 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":352,"mutex_wait_us":5,"thread_start_us":93,"threads_started":1}
I20260929 17:53:34.136895 7862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:34.132829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43346 (local address 127.0.98.1:46193)
0929 17:53:34.133305 (+ 476us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:34.133310 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:34.133324 (+ 14us) server_negotiation.cc:407] Connection header received
0929 17:53:34.133374 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:34.133378 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:34.133427 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:34.133511 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:53:34.133892 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:34.134483 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:34.135741 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:34.136307 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:34.136398 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:34.136460 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0929 17:53:34.136709 (+ 249us) server_negotiation.cc:299] Negotiation successful
0929 17:53:34.136783 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":392,"thread_start_us":70,"threads_started":1}
I20260929 17:53:34.137064 7695 tablet_service.cc:1467] Processing CreateTablet for tablet 74a2a84d041f41acb717f0d88091d045 (DEFAULT_TABLE table=test-table [id=71f6b06b12e84e379ce1c58d8fe6941c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 17:53:34.137382 7695 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74a2a84d041f41acb717f0d88091d045. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:53:34.137579 7420 tablet_service.cc:1467] Processing CreateTablet for tablet 74a2a84d041f41acb717f0d88091d045 (DEFAULT_TABLE table=test-table [id=71f6b06b12e84e379ce1c58d8fe6941c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 17:53:34.137832 7420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74a2a84d041f41acb717f0d88091d045. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:53:34.139714 7865 tablet_bootstrap.cc:492] T 74a2a84d041f41acb717f0d88091d045 P c4d0d8b2d8994344a6ec327d5c389bba: Bootstrap starting.
I20260929 17:53:34.140198 7866 tablet_bootstrap.cc:492] T 74a2a84d041f41acb717f0d88091d045 P 7cb83ff6a04043169a3cf2e232462596: Bootstrap starting.
I20260929 17:53:34.140767 7865 tablet_bootstrap.cc:654] T 74a2a84d041f41acb717f0d88091d045 P c4d0d8b2d8994344a6ec327d5c389bba: Neither blocks nor log segments found. Creating new log.
I20260929 17:53:34.141047 7866 tablet_bootstrap.cc:654] T 74a2a84d041f41acb717f0d88091d045 P 7cb83ff6a04043169a3cf2e232462596: Neither blocks nor log segments found. Creating new log.
I20260929 17:53:34.141781 7864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:34.133536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49778 (local address 127.0.98.2:43951)
0929 17:53:34.134018 (+ 482us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:34.134023 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:34.134048 (+ 25us) server_negotiation.cc:407] Connection header received
0929 17:53:34.134705 (+ 657us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:34.134709 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:34.134764 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:34.134846 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:53:34.135109 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:34.135769 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:34.140529 (+ 4760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:34.141151 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:34.141272 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:34.141339 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0929 17:53:34.141589 (+ 250us) server_negotiation.cc:299] Negotiation successful
0929 17:53:34.141668 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":406,"thread_start_us":90,"threads_started":1}
I20260929 17:53:34.141968 7865 tablet_bootstrap.cc:492] T 74a2a84d041f41acb717f0d88091d045 P c4d0d8b2d8994344a6ec327d5c389bba: No bootstrap required, opened a new log
I20260929 17:53:34.142055 7865 ts_tablet_manager.cc:1397] T 74a2a84d041f41acb717f0d88091d045 P c4d0d8b2d8994344a6ec327d5c389bba: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20260929 17:53:34.142186 7863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:34.133139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43951 (local address 127.0.0.1:49778)
0929 17:53:34.133713 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:34.133726 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 17:53:34.133804 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:53:34.134865 (+ 1061us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:53:34.134869 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:53:34.134889 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:53:34.134995 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:34.135002 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:34.135867 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:34.135870 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:34.140378 (+ 4508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:34.140390 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:34.141287 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:34.141292 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:34.141410 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:34.141440 (+ 30us) client_negotiation.cc:769] Sending connection context
0929 17:53:34.141937 (+ 497us) client_negotiation.cc:240] Negotiation successful
0929 17:53:34.142019 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":519,"thread_start_us":87,"threads_started":1}
I20260929 17:53:34.142472 7865 raft_consensus.cc:348] T 74a2a84d041f41acb717f0d88091d045 P c4d0d8b2d8994344a6ec327d5c389bba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4d0d8b2d8994344a6ec327d5c389bba" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41685 } } peers { permanent_uuid: "ade19fb6904045318b25a43744e7c3f3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43951 } } peers { permanent_uuid: "7cb83ff6a04043169a3cf2e232462596" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46193 } }
I20260929 17:53:34.142578 7865 raft_consensus.cc:374] T 74a2a84d041f41acb717f0d88091d045 P c4d0d8b2d8994344a6ec327d5c389bba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:53:34.142608 7865 raft_consensus.cc:729] T 74a2a84d041f41acb717f0d88091d045 P c4d0d8b2d8994344a6ec327d5c389bba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4d0d8b2d8994344a6ec327d5c389bba, State: Initialized, Role: FOLLOWER
I20260929 17:53:34.142738 7865 consensus_queue.cc:260] T 74a2a84d041f41acb717f0d88091d045 P c4d0d8b2d8994344a6ec327d5c389bba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4d0d8b2d8994344a6ec327d5c389bba" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41685 } } peers { permanent_uuid: "ade19fb6904045318b25a43744e7c3f3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43951 } } peers { permanent_uuid: "7cb83ff6a04043169a3cf2e232462596" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46193 } }
I20260929 17:53:34.142768 7557 tablet_service.cc:1467] Processing CreateTablet for tablet 74a2a84d041f41acb717f0d88091d045 (DEFAULT_TABLE table=test-table [id=71f6b06b12e84e379ce1c58d8fe6941c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 17:53:34.143059 7557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74a2a84d041f41acb717f0d88091d045. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:53:34.143105 7865 ts_tablet_manager.cc:1428] T 74a2a84d041f41acb717f0d88091d045 P c4d0d8b2d8994344a6ec327d5c389bba: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260929 17:53:34.144796 7846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:34.144551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39863 (local address 127.0.98.3:41223)
0929 17:53:34.144660 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:34.144724 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260929 17:53:34.145094 7866 tablet_bootstrap.cc:492] T 74a2a84d041f41acb717f0d88091d045 P 7cb83ff6a04043169a3cf2e232462596: No bootstrap required, opened a new log
I20260929 17:53:34.145195 7866 ts_tablet_manager.cc:1397] T 74a2a84d041f41acb717f0d88091d045 P 7cb83ff6a04043169a3cf2e232462596: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.001s
I20260929 17:53:34.146013 7763 heartbeater.cc:503] Master 127.0.98.61:34183 was elected leader, sending a full tablet report...
I20260929 17:53:34.146219 7870 tablet_bootstrap.cc:492] T 74a2a84d041f41acb717f0d88091d045 P ade19fb6904045318b25a43744e7c3f3: Bootstrap starting.
I20260929 17:53:34.146085 7866 raft_consensus.cc:348] T 74a2a84d041f41acb717f0d88091d045 P 7cb83ff6a04043169a3cf2e232462596 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4d0d8b2d8994344a6ec327d5c389bba" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41685 } } peers { permanent_uuid: "ade19fb6904045318b25a43744e7c3f3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43951 } } peers { permanent_uuid: "7cb83ff6a04043169a3cf2e232462596" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46193 } }
I20260929 17:53:34.146298 7866 raft_consensus.cc:374] T 74a2a84d041f41acb717f0d88091d045 P 7cb83ff6a04043169a3cf2e232462596 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:53:34.146351 7866 raft_consensus.cc:729] T 74a2a84d041f41acb717f0d88091d045 P 7cb83ff6a04043169a3cf2e232462596 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7cb83ff6a04043169a3cf2e232462596, State: Initialized, Role: FOLLOWER
I20260929 17:53:34.146505 7866 consensus_queue.cc:260] T 74a2a84d041f41acb717f0d88091d045 P 7cb83ff6a04043169a3cf2e232462596 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4d0d8b2d8994344a6ec327d5c389bba" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41685 } } peers { permanent_uuid: "ade19fb6904045318b25a43744e7c3f3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43951 } } peers { permanent_uuid: "7cb83ff6a04043169a3cf2e232462596" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46193 } }
I20260929 17:53:34.146914 7866 ts_tablet_manager.cc:1428] T 74a2a84d041f41acb717f0d88091d045 P 7cb83ff6a04043169a3cf2e232462596: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260929 17:53:34.147570 7870 tablet_bootstrap.cc:654] T 74a2a84d041f41acb717f0d88091d045 P ade19fb6904045318b25a43744e7c3f3: Neither blocks nor log segments found. Creating new log.
I20260929 17:53:34.147920 7845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:34.147666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39863 (local address 127.0.98.1:49891)
0929 17:53:34.147791 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:34.147847 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260929 17:53:34.148962 7845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:34.148717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39863 (local address 127.0.98.1:56145)
0929 17:53:34.148849 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:34.148898 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260929 17:53:34.149194 7870 tablet_bootstrap.cc:492] T 74a2a84d041f41acb717f0d88091d045 P ade19fb6904045318b25a43744e7c3f3: No bootstrap required, opened a new log
I20260929 17:53:34.149281 7870 ts_tablet_manager.cc:1397] T 74a2a84d041f41acb717f0d88091d045 P ade19fb6904045318b25a43744e7c3f3: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260929 17:53:34.149717 7488 heartbeater.cc:503] Master 127.0.98.61:34183 was elected leader, sending a full tablet report...
I20260929 17:53:34.149746 7870 raft_consensus.cc:348] T 74a2a84d041f41acb717f0d88091d045 P ade19fb6904045318b25a43744e7c3f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4d0d8b2d8994344a6ec327d5c389bba" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41685 } } peers { permanent_uuid: "ade19fb6904045318b25a43744e7c3f3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43951 } } peers { permanent_uuid: "7cb83ff6a04043169a3cf2e232462596" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46193 } }
I20260929 17:53:34.149863 7870 raft_consensus.cc:374] T 74a2a84d041f41acb717f0d88091d045 P ade19fb6904045318b25a43744e7c3f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:53:34.149904 7870 raft_consensus.cc:729] T 74a2a84d041f41acb717f0d88091d045 P ade19fb6904045318b25a43744e7c3f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ade19fb6904045318b25a43744e7c3f3, State: Initialized, Role: FOLLOWER
I20260929 17:53:34.150054 7870 consensus_queue.cc:260] T 74a2a84d041f41acb717f0d88091d045 P ade19fb6904045318b25a43744e7c3f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4d0d8b2d8994344a6ec327d5c389bba" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41685 } } peers { permanent_uuid: "ade19fb6904045318b25a43744e7c3f3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43951 } } peers { permanent_uuid: "7cb83ff6a04043169a3cf2e232462596" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46193 } }
I20260929 17:53:34.150442 7870 ts_tablet_manager.cc:1428] T 74a2a84d041f41acb717f0d88091d045 P ade19fb6904045318b25a43744e7c3f3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 17:53:34.151002 7844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:34.150716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39863 (local address 127.0.98.2:44811)
0929 17:53:34.150874 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:34.150931 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260929 17:53:34.151721 7844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:34.151452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39863 (local address 127.0.98.2:44963)
0929 17:53:34.151607 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:34.151658 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
I20260929 17:53:34.152032 7625 heartbeater.cc:503] Master 127.0.98.61:34183 was elected leader, sending a full tablet report...
W20260929 17:53:34.220804 7764 tablet.cc:2369] T 74a2a84d041f41acb717f0d88091d045 P c4d0d8b2d8994344a6ec327d5c389bba: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 17:53:34.262554 7626 tablet.cc:2369] T 74a2a84d041f41acb717f0d88091d045 P ade19fb6904045318b25a43744e7c3f3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 17:53:34.312644 7869 raft_consensus.cc:482] T 74a2a84d041f41acb717f0d88091d045 P c4d0d8b2d8994344a6ec327d5c389bba [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 17:53:34.312808 7869 raft_consensus.cc:504] T 74a2a84d041f41acb717f0d88091d045 P c4d0d8b2d8994344a6ec327d5c389bba [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4d0d8b2d8994344a6ec327d5c389bba" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41685 } } peers { permanent_uuid: "ade19fb6904045318b25a43744e7c3f3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43951 } } peers { permanent_uuid: "7cb83ff6a04043169a3cf2e232462596" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46193 } }
I20260929 17:53:34.313938 7869 leader_election.cc:288] T 74a2a84d041f41acb717f0d88091d045 P c4d0d8b2d8994344a6ec327d5c389bba [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ade19fb6904045318b25a43744e7c3f3 (127.0.98.2:43951), 7cb83ff6a04043169a3cf2e232462596 (127.0.98.1:46193)
I20260929 17:53:34.318295 7862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:34.314292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40841 (local address 127.0.98.1:46193)
0929 17:53:34.314562 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:34.314566 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:34.314581 (+ 15us) server_negotiation.cc:407] Connection header received
0929 17:53:34.314631 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:34.314633 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:34.314678 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:34.314777 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:53:34.315044 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:34.315756 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:34.317357 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:34.317755 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:34.317854 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:34.317923 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0929 17:53:34.318115 (+ 192us) server_negotiation.cc:299] Negotiation successful
0929 17:53:34.318192 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":199}
I20260929 17:53:34.318295 7846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:34.314192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46193 (local address 127.0.98.3:40841)
0929 17:53:34.314295 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:34.314310 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 17:53:34.314415 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:53:34.314786 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:53:34.314789 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:53:34.314808 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:53:34.314928 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:34.314934 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:34.315874 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:34.315876 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:34.317242 (+ 1366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:34.317252 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:34.317896 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:34.317900 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:34.317993 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:34.318014 (+ 21us) client_negotiation.cc:769] Sending connection context
0929 17:53:34.318084 (+ 70us) client_negotiation.cc:240] Negotiation successful
0929 17:53:34.318151 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25,"mutex_wait_us":19}
I20260929 17:53:34.318703 7874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:34.314196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43951 (local address 127.0.98.3:49389)
0929 17:53:34.314616 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:34.314626 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 17:53:34.314707 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:53:34.315267 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:53:34.315270 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:53:34.315292 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:53:34.315397 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:34.315402 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:34.316228 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:34.316231 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:34.317562 (+ 1331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:34.317572 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:34.318179 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:34.318183 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:34.318278 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:34.318299 (+ 21us) client_negotiation.cc:769] Sending connection context
0929 17:53:34.318506 (+ 207us) client_negotiation.cc:240] Negotiation successful
0929 17:53:34.318575 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":362,"mutex_wait_us":152,"thread_start_us":134,"threads_started":1}
I20260929 17:53:34.318745 7864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:34.314428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49389 (local address 127.0.98.2:43951)
0929 17:53:34.314547 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:34.314551 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:34.314699 (+ 148us) server_negotiation.cc:407] Connection header received
0929 17:53:34.315070 (+ 371us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:34.315073 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:34.315123 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:34.315196 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:53:34.315516 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:34.316127 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:34.317676 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:34.318068 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:34.318161 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:34.318218 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0929 17:53:34.318537 (+ 319us) server_negotiation.cc:299] Negotiation successful
0929 17:53:34.318623 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 17:53:34.318928 7440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74a2a84d041f41acb717f0d88091d045" candidate_uuid: "c4d0d8b2d8994344a6ec327d5c389bba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7cb83ff6a04043169a3cf2e232462596" is_pre_election: true
I20260929 17:53:34.319188 7440 raft_consensus.cc:2454] T 74a2a84d041f41acb717f0d88091d045 P 7cb83ff6a04043169a3cf2e232462596 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c4d0d8b2d8994344a6ec327d5c389bba in term 0.
I20260929 17:53:34.319276 7577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74a2a84d041f41acb717f0d88091d045" candidate_uuid: "c4d0d8b2d8994344a6ec327d5c389bba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ade19fb6904045318b25a43744e7c3f3" is_pre_election: true
I20260929 17:53:34.319412 7577 raft_consensus.cc:2454] T 74a2a84d041f41acb717f0d88091d045 P ade19fb6904045318b25a43744e7c3f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c4d0d8b2d8994344a6ec327d5c389bba in term 0.
I20260929 17:53:34.319583 7647 leader_election.cc:302] T 74a2a84d041f41acb717f0d88091d045 P c4d0d8b2d8994344a6ec327d5c389bba [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7cb83ff6a04043169a3cf2e232462596, c4d0d8b2d8994344a6ec327d5c389bba; no voters:
I20260929 17:53:34.319864 7869 raft_consensus.cc:2781] T 74a2a84d041f41acb717f0d88091d045 P c4d0d8b2d8994344a6ec327d5c389bba [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 17:53:34.319921 7869 raft_consensus.cc:482] T 74a2a84d041f41acb717f0d88091d045 P c4d0d8b2d8994344a6ec327d5c389bba [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 17:53:34.319947 7869 raft_consensus.cc:3037] T 74a2a84d041f41acb717f0d88091d045 P c4d0d8b2d8994344a6ec327d5c389bba [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:53:34.320952 7869 raft_consensus.cc:504] T 74a2a84d041f41acb717f0d88091d045 P c4d0d8b2d8994344a6ec327d5c389bba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4d0d8b2d8994344a6ec327d5c389bba" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41685 } } peers { permanent_uuid: "ade19fb6904045318b25a43744e7c3f3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43951 } } peers { permanent_uuid: "7cb83ff6a04043169a3cf2e232462596" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46193 } }
I20260929 17:53:34.321322 7869 leader_election.cc:288] T 74a2a84d041f41acb717f0d88091d045 P c4d0d8b2d8994344a6ec327d5c389bba [CANDIDATE]: Term 1 election: Requested vote from peers ade19fb6904045318b25a43744e7c3f3 (127.0.98.2:43951), 7cb83ff6a04043169a3cf2e232462596 (127.0.98.1:46193)
I20260929 17:53:34.321635 7440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74a2a84d041f41acb717f0d88091d045" candidate_uuid: "c4d0d8b2d8994344a6ec327d5c389bba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7cb83ff6a04043169a3cf2e232462596"
I20260929 17:53:34.321635 7577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74a2a84d041f41acb717f0d88091d045" candidate_uuid: "c4d0d8b2d8994344a6ec327d5c389bba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ade19fb6904045318b25a43744e7c3f3"
I20260929 17:53:34.321753 7577 raft_consensus.cc:3037] T 74a2a84d041f41acb717f0d88091d045 P ade19fb6904045318b25a43744e7c3f3 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:53:34.321753 7440 raft_consensus.cc:3037] T 74a2a84d041f41acb717f0d88091d045 P 7cb83ff6a04043169a3cf2e232462596 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:53:34.322870 7577 raft_consensus.cc:2454] T 74a2a84d041f41acb717f0d88091d045 P ade19fb6904045318b25a43744e7c3f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c4d0d8b2d8994344a6ec327d5c389bba in term 1.
I20260929 17:53:34.322870 7440 raft_consensus.cc:2454] T 74a2a84d041f41acb717f0d88091d045 P 7cb83ff6a04043169a3cf2e232462596 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c4d0d8b2d8994344a6ec327d5c389bba in term 1.
I20260929 17:53:34.323112 7647 leader_election.cc:302] T 74a2a84d041f41acb717f0d88091d045 P c4d0d8b2d8994344a6ec327d5c389bba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7cb83ff6a04043169a3cf2e232462596, c4d0d8b2d8994344a6ec327d5c389bba; no voters:
I20260929 17:53:34.323263 7869 raft_consensus.cc:2781] T 74a2a84d041f41acb717f0d88091d045 P c4d0d8b2d8994344a6ec327d5c389bba [term 1 FOLLOWER]: Leader election won for term 1
I20260929 17:53:34.323446 7869 raft_consensus.cc:686] T 74a2a84d041f41acb717f0d88091d045 P c4d0d8b2d8994344a6ec327d5c389bba [term 1 LEADER]: Becoming Leader. State: Replica: c4d0d8b2d8994344a6ec327d5c389bba, State: Running, Role: LEADER
I20260929 17:53:34.323858 7869 consensus_queue.cc:237] T 74a2a84d041f41acb717f0d88091d045 P c4d0d8b2d8994344a6ec327d5c389bba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4d0d8b2d8994344a6ec327d5c389bba" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41685 } } peers { permanent_uuid: "ade19fb6904045318b25a43744e7c3f3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43951 } } peers { permanent_uuid: "7cb83ff6a04043169a3cf2e232462596" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46193 } }
I20260929 17:53:34.324153 7874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:34.323897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39863 (local address 127.0.98.3:50185)
0929 17:53:34.324037 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:34.324092 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260929 17:53:34.326154 7122 catalog_manager.cc:5495] T 74a2a84d041f41acb717f0d88091d045 P c4d0d8b2d8994344a6ec327d5c389bba reported cstate change: term changed from 0 to 1, leader changed from <none> to c4d0d8b2d8994344a6ec327d5c389bba (127.0.98.3). New cstate: current_term: 1 leader_uuid: "c4d0d8b2d8994344a6ec327d5c389bba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4d0d8b2d8994344a6ec327d5c389bba" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41685 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ade19fb6904045318b25a43744e7c3f3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43951 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7cb83ff6a04043169a3cf2e232462596" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46193 } health_report { overall_health: UNKNOWN } } }
I20260929 17:53:34.375988 7860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:34.373021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56258 (local address 127.0.98.3:41685)
0929 17:53:34.373132 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:34.373135 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:34.373149 (+ 14us) server_negotiation.cc:407] Connection header received
0929 17:53:34.373230 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:34.373232 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:34.373286 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:34.373372 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 17:53:34.373543 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:34.374053 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:34.374811 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:34.375011 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:34.375077 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:34.375270 (+ 193us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 17:53:34.375705 (+ 435us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 17:53:34.375786 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0929 17:53:34.375830 (+ 44us) server_negotiation.cc:299] Negotiation successful
0929 17:53:34.375874 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260929 17:53:34.378706 7577 raft_consensus.cc:1261] T 74a2a84d041f41acb717f0d88091d045 P ade19fb6904045318b25a43744e7c3f3 [term 1 FOLLOWER]: Refusing update from remote peer c4d0d8b2d8994344a6ec327d5c389bba: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 17:53:34.378896 7440 raft_consensus.cc:1261] T 74a2a84d041f41acb717f0d88091d045 P 7cb83ff6a04043169a3cf2e232462596 [term 1 FOLLOWER]: Refusing update from remote peer c4d0d8b2d8994344a6ec327d5c389bba: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 17:53:34.379182 7875 consensus_queue.cc:1035] T 74a2a84d041f41acb717f0d88091d045 P c4d0d8b2d8994344a6ec327d5c389bba [LEADER]: Connected to new peer: Peer: permanent_uuid: "ade19fb6904045318b25a43744e7c3f3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 17:53:34.379448 7869 consensus_queue.cc:1035] T 74a2a84d041f41acb717f0d88091d045 P c4d0d8b2d8994344a6ec327d5c389bba [LEADER]: Connected to new peer: Peer: permanent_uuid: "7cb83ff6a04043169a3cf2e232462596" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 17:53:34.380755 7844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:34.380473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39863 (local address 127.0.98.2:42951)
0929 17:53:34.380614 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:34.380670 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260929 17:53:34.381301 7845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:34.381031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39863 (local address 127.0.98.1:59325)
0929 17:53:34.381171 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:34.381234 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260929 17:53:34.383351 7878 mvcc.cc:205] Tried to move back new op lower bound from 7334725281290424320 to 7334725281073008640. Current Snapshot: MvccSnapshot[applied={T|T < 7334725281290424320}]
I20260929 17:53:34.384199 7874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:34.383945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39863 (local address 127.0.98.3:56063)
0929 17:53:34.384082 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:34.384132 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260929 17:53:34.384284 7879 mvcc.cc:205] Tried to move back new op lower bound from 7334725281290424320 to 7334725281073008640. Current Snapshot: MvccSnapshot[applied={T|T < 7334725281290424320}]
I20260929 17:53:34.384397 7844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:34.384112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39863 (local address 127.0.98.2:34159)
0929 17:53:34.384291 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:34.384339 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118}
I20260929 17:53:34.385854 7845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:34.385628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39863 (local address 127.0.98.1:50843)
0929 17:53:34.385751 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:34.385795 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260929 17:53:34.387665 7874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:34.387454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39863 (local address 127.0.98.3:39295)
0929 17:53:34.387559 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:34.387605 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260929 17:53:34.392828 7122 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41528:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 17:53:34.393023 7122 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41528:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 17:53:34.396538 7122 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 952a9b1a252747a8acdcfa8e04debe30: Sending DeleteTablet for 3 replicas of tablet 74a2a84d041f41acb717f0d88091d045
I20260929 17:53:34.397357 7557 tablet_service.cc:1514] Processing DeleteTablet for tablet 74a2a84d041f41acb717f0d88091d045 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 17:53:34 UTC) from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:49778
I20260929 17:53:34.397553 7420 tablet_service.cc:1514] Processing DeleteTablet for tablet 74a2a84d041f41acb717f0d88091d045 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 17:53:34 UTC) from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:43346
I20260929 17:53:34.397660 7695 tablet_service.cc:1514] Processing DeleteTablet for tablet 74a2a84d041f41acb717f0d88091d045 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 17:53:34 UTC) from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:56246
I20260929 17:53:34.397719 7888 tablet_replica.cc:330] T 74a2a84d041f41acb717f0d88091d045 P ade19fb6904045318b25a43744e7c3f3: stopping tablet replica
I20260929 17:53:34.397847 7888 raft_consensus.cc:2229] T 74a2a84d041f41acb717f0d88091d045 P ade19fb6904045318b25a43744e7c3f3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 17:53:34.397876 7889 tablet_replica.cc:330] T 74a2a84d041f41acb717f0d88091d045 P 7cb83ff6a04043169a3cf2e232462596: stopping tablet replica
I20260929 17:53:34.397962 7889 raft_consensus.cc:2229] T 74a2a84d041f41acb717f0d88091d045 P 7cb83ff6a04043169a3cf2e232462596 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 17:53:34.397980 7888 raft_consensus.cc:2258] T 74a2a84d041f41acb717f0d88091d045 P ade19fb6904045318b25a43744e7c3f3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 17:53:34.398077 7889 raft_consensus.cc:2258] T 74a2a84d041f41acb717f0d88091d045 P 7cb83ff6a04043169a3cf2e232462596 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 17:53:34.398089 7890 tablet_replica.cc:330] T 74a2a84d041f41acb717f0d88091d045 P c4d0d8b2d8994344a6ec327d5c389bba: stopping tablet replica
I20260929 17:53:34.398264 7890 raft_consensus.cc:2229] T 74a2a84d041f41acb717f0d88091d045 P c4d0d8b2d8994344a6ec327d5c389bba [term 1 LEADER]: Raft consensus shutting down.
I20260929 17:53:34.398464 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskup3Soe/build/debug/bin/kudu with pid 7268
I20260929 17:53:34.398640 7890 raft_consensus.cc:2258] T 74a2a84d041f41acb717f0d88091d045 P c4d0d8b2d8994344a6ec327d5c389bba [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 17:53:34.399474 7888 ts_tablet_manager.cc:1905] T 74a2a84d041f41acb717f0d88091d045 P ade19fb6904045318b25a43744e7c3f3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 17:53:34.399909 7890 ts_tablet_manager.cc:1905] T 74a2a84d041f41acb717f0d88091d045 P c4d0d8b2d8994344a6ec327d5c389bba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 17:53:34.401917 7888 ts_tablet_manager.cc:1918] T 74a2a84d041f41acb717f0d88091d045 P ade19fb6904045318b25a43744e7c3f3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 17:53:34.401995 7888 log.cc:1196] T 74a2a84d041f41acb717f0d88091d045 P ade19fb6904045318b25a43744e7c3f3: Deleting WAL directory at /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704375975685-392-0/minicluster-data/ts-1/wal/wals/74a2a84d041f41acb717f0d88091d045
I20260929 17:53:34.402238 7888 ts_tablet_manager.cc:1939] T 74a2a84d041f41acb717f0d88091d045 P ade19fb6904045318b25a43744e7c3f3: Deleting consensus metadata
I20260929 17:53:34.402565 7890 ts_tablet_manager.cc:1918] T 74a2a84d041f41acb717f0d88091d045 P c4d0d8b2d8994344a6ec327d5c389bba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 17:53:34.402690 7890 log.cc:1196] T 74a2a84d041f41acb717f0d88091d045 P c4d0d8b2d8994344a6ec327d5c389bba: Deleting WAL directory at /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790704375975685-392-0/minicluster-data/ts-2/wal/wals/74a2a84d041f41acb717f0d88091d045
I20260929 17:53:34.402871 7110 catalog_manager.cc:4841] TS ade19fb6904045318b25a43744e7c3f3 (127.0.98.2:43951): tablet 74a2a84d041f41acb717f0d88091d045 (table test-table [id=71f6b06b12e84e379ce1c58d8fe6941c]) successfully deleted
I20260929 17:53:34.403040 7890 ts_tablet_manager.cc:1939] T 74a2a84d041f41acb717f0d88091d045 P c4d0d8b2d8994344a6ec327d5c389bba: Deleting consensus metadata
I20260929 17:53:34.403698 7107 catalog_manager.cc:4841] TS c4d0d8b2d8994344a6ec327d5c389bba (127.0.98.3:41685): tablet 74a2a84d041f41acb717f0d88091d045 (table test-table [id=71f6b06b12e84e379ce1c58d8fe6941c]) successfully deleted
I20260929 17:53:34.406894 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskup3Soe/build/debug/bin/kudu with pid 7494
I20260929 17:53:34.406980 7863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:34.406729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46193 (local address 127.0.0.1:43362)
0929 17:53:34.406849 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:34.406901 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260929 17:53:34.408942 7107 catalog_manager.cc:4584] TS 7cb83ff6a04043169a3cf2e232462596 (127.0.98.1:46193): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 74a2a84d041f41acb717f0d88091d045: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46193: connect: Connection refused (error 111)
I20260929 17:53:34.413327 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskup3Soe/build/debug/bin/kudu with pid 7630
I20260929 17:53:34.418375 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskup3Soe/build/debug/bin/kudu with pid 6987
I20260929 17:53:34.423219 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskup3Soe/build/debug/bin/kudu with pid 7095
2026-09-29T17:53:34Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (7095 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704375975685-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 17:53:34 dist-test-slave-53nh krb5kdc[7893](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 17:53:34 dist-test-slave-53nh krb5kdc[7893](info): set up 2 sockets
Sep 29 17:53:34 dist-test-slave-53nh krb5kdc[7893](info): commencing operation
krb5kdc: starting...
W20260929 17:53:35.377326 392 mini_kdc.cc:121] Time spent starting KDC: real 0.933s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704375975685-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704375975685-392-0/krb5kdc/test-user.keytab.
Sep 29 17:53:35 dist-test-slave-53nh krb5kdc[7893](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704415, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T17:53:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T17:53:35Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704375975685-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704375975685-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260929 17:53:35.491397 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskup3Soe/build/debug/bin/kudu
/tmp/dist-test-taskup3Soe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704375975685-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704375975685-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704375975685-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704375975685-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44965
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704375975685-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44965
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704375975685-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704375975685-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704375975685-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:53:35.599926 7906 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:53:35.600206 7906 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:53:35.600262 7906 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:53:35.604233 7906 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 17:53:35.604301 7906 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260929 17:53:35.604324 7906 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:53:35.604347 7906 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:53:35.604393 7906 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 17:53:35.604421 7906 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 17:53:35.609794 7906 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704375975685-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704375975685-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44965
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704375975685-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44965
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704375975685-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7906
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704375975685-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:53:35.610879 7906 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:53:35.611907 7906 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:53:35.620113 7914 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:53:35.620239 7911 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:53:35.620334 7906 server_base.cc:1034] running on GCE node
W20260929 17:53:35.620113 7912 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:53:35.620975 7906 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:53:35.622093 7906 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:53:35.623246 7906 hybrid_clock.cc:648] HybridClock initialized: now 1790704415623226 us; error 34 us; skew 500 ppm
Sep 29 17:53:35 dist-test-slave-53nh krb5kdc[7893](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704415, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:53:35.625988 7906 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260929 17:53:35.626863 7906 webserver.cc:460] Webserver started at http://127.0.98.62:33883/ using document root <none> and password file <none>
I20260929 17:53:35.627403 7906 fs_manager.cc:359] Metadata directory not provided
I20260929 17:53:35.627460 7906 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:53:35.627638 7906 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:53:35.629695 7906 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704375975685-392-0/minicluster-data/master-0/data/instance:
uuid: "1fb78915f2ed49e5b241da7350b145ae"
format_stamp: "Formatted at 2026-09-29 17:53:35 on dist-test-slave-53nh"
server_key: "6df5feb12ecfb08c8a7c54c226a470b3"
server_key_iv: "6bd22fb8affed6e0494535ff39d0513e"
server_key_version: "encryptionkey@0"
I20260929 17:53:35.630223 7906 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704375975685-392-0/minicluster-data/master-0/wal/instance:
uuid: "1fb78915f2ed49e5b241da7350b145ae"
format_stamp: "Formatted at 2026-09-29 17:53:35 on dist-test-slave-53nh"
server_key: "6df5feb12ecfb08c8a7c54c226a470b3"
server_key_iv: "6bd22fb8affed6e0494535ff39d0513e"
server_key_version: "encryptionkey@0"
I20260929 17:53:35.633754 7906 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 17:53:35.636019 7921 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:53:35.637240 7906 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 17:53:35.637341 7906 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704375975685-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704375975685-392-0/minicluster-data/master-0/wal
uuid: "1fb78915f2ed49e5b241da7350b145ae"
format_stamp: "Formatted at 2026-09-29 17:53:35 on dist-test-slave-53nh"
server_key: "6df5feb12ecfb08c8a7c54c226a470b3"
server_key_iv: "6bd22fb8affed6e0494535ff39d0513e"
server_key_version: "encryptionkey@0"
I20260929 17:53:35.637435 7906 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704375975685-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704375975685-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704375975685-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:53:35.652823 7906 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:53:35.655519 7906 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:53:35.655689 7906 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:53:35.662885 7906 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44965
I20260929 17:53:35.662910 7983 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44965 every 8 connection(s)
I20260929 17:53:35.663769 7906 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704375975685-392-0/minicluster-data/master-0/data/info.pb
I20260929 17:53:35.666463 7984 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:53:35.667227 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskup3Soe/build/debug/bin/kudu as pid 7906
I20260929 17:53:35.667353 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704375975685-392-0/minicluster-data/master-0/wal/instance
I20260929 17:53:35.667651 392 external_mini_cluster.cc:1428] Setting key 47dfd49b04e59aa6a0567ee80c8e5a99
I20260929 17:53:35.672257 7984 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1fb78915f2ed49e5b241da7350b145ae: Bootstrap starting.
I20260929 17:53:35.674499 7984 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1fb78915f2ed49e5b241da7350b145ae: Neither blocks nor log segments found. Creating new log.
I20260929 17:53:35.675184 7984 log.cc:824] T 00000000000000000000000000000000 P 1fb78915f2ed49e5b241da7350b145ae: Log is configured to *not* fsync() on all Append() calls
Sep 29 17:53:35 dist-test-slave-53nh krb5kdc[7893](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704415, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260929 17:53:35.677245 7984 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1fb78915f2ed49e5b241da7350b145ae: No bootstrap required, opened a new log
I20260929 17:53:35.680481 7984 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1fb78915f2ed49e5b241da7350b145ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fb78915f2ed49e5b241da7350b145ae" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44965 } }
I20260929 17:53:35.680686 7984 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1fb78915f2ed49e5b241da7350b145ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:53:35.680728 7984 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1fb78915f2ed49e5b241da7350b145ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1fb78915f2ed49e5b241da7350b145ae, State: Initialized, Role: FOLLOWER
I20260929 17:53:35.681154 7984 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1fb78915f2ed49e5b241da7350b145ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fb78915f2ed49e5b241da7350b145ae" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44965 } }
I20260929 17:53:35.681200 7987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:35.669088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51916 (local address 127.0.98.62:44965)
0929 17:53:35.669493 (+ 405us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:35.669507 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:35.669541 (+ 34us) server_negotiation.cc:407] Connection header received
0929 17:53:35.670149 (+ 608us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:35.670164 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:35.670484 (+ 320us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:35.670774 (+ 290us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:53:35.671533 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:35.672518 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:35.673370 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:35.673826 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:35.673980 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:35.676425 (+ 2445us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:53:35.676429 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:53:35.676441 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:53:35.676465 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:53:35.678531 (+ 2066us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:53:35.678989 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:53:35.678993 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:53:35.678997 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:53:35.679106 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:53:35.679354 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:53:35.679356 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:53:35.679357 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:53:35.679703 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:53:35.679841 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0929 17:53:35.680106 (+ 265us) server_negotiation.cc:299] Negotiation successful
0929 17:53:35.680399 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":124,"threads_started":1}
I20260929 17:53:35.681291 7984 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1fb78915f2ed49e5b241da7350b145ae [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 17:53:35.681346 7984 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1fb78915f2ed49e5b241da7350b145ae [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 17:53:35.681460 7984 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1fb78915f2ed49e5b241da7350b145ae [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:53:35.682315 7984 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1fb78915f2ed49e5b241da7350b145ae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fb78915f2ed49e5b241da7350b145ae" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44965 } }
I20260929 17:53:35.682652 7984 leader_election.cc:302] T 00000000000000000000000000000000 P 1fb78915f2ed49e5b241da7350b145ae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1fb78915f2ed49e5b241da7350b145ae; no voters:
I20260929 17:53:35.682948 7984 leader_election.cc:288] T 00000000000000000000000000000000 P 1fb78915f2ed49e5b241da7350b145ae [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 17:53:35.683025 7989 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1fb78915f2ed49e5b241da7350b145ae [term 1 FOLLOWER]: Leader election won for term 1
I20260929 17:53:35.683508 7989 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1fb78915f2ed49e5b241da7350b145ae [term 1 LEADER]: Becoming Leader. State: Replica: 1fb78915f2ed49e5b241da7350b145ae, State: Running, Role: LEADER
I20260929 17:53:35.683817 7989 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1fb78915f2ed49e5b241da7350b145ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fb78915f2ed49e5b241da7350b145ae" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44965 } }
I20260929 17:53:35.684077 7984 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1fb78915f2ed49e5b241da7350b145ae [sys.catalog]: configured and running, proceeding with master startup.
I20260929 17:53:35.685398 7991 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1fb78915f2ed49e5b241da7350b145ae [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1fb78915f2ed49e5b241da7350b145ae. Latest consensus state: current_term: 1 leader_uuid: "1fb78915f2ed49e5b241da7350b145ae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fb78915f2ed49e5b241da7350b145ae" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44965 } } }
I20260929 17:53:35.685390 7990 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1fb78915f2ed49e5b241da7350b145ae [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1fb78915f2ed49e5b241da7350b145ae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fb78915f2ed49e5b241da7350b145ae" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44965 } } }
I20260929 17:53:35.685516 7991 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1fb78915f2ed49e5b241da7350b145ae [sys.catalog]: This master's current role is: LEADER
I20260929 17:53:35.685516 7990 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1fb78915f2ed49e5b241da7350b145ae [sys.catalog]: This master's current role is: LEADER
I20260929 17:53:35.686239 8003 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 17:53:35.688800 8003 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 17:53:35.693959 8003 catalog_manager.cc:1347] Generated new cluster ID: 1ede8f015d034a7f8d780cdad497716f
I20260929 17:53:35.694013 8003 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 17:53:35.706818 8003 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 17:53:35.708062 8003 catalog_manager.cc:1504] Loading token signing keys...
I20260929 17:53:35.716190 8003 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1fb78915f2ed49e5b241da7350b145ae: Generated new TSK 0
I20260929 17:53:35.716897 8003 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704375975685-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704375975685-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260929 17:53:35.747967 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskup3Soe/build/debug/bin/kudu
/tmp/dist-test-taskup3Soe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704375975685-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704375975685-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704375975685-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704375975685-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44965
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704375975685-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704375975685-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704375975685-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704375975685-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:53:35.855329 8010 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:53:35.855623 8010 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:53:35.855680 8010 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:53:35.859771 8010 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:53:35.859853 8010 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:53:35.859897 8010 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 17:53:35.859954 8010 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 17:53:35.865347 8010 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704375975685-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704375975685-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704375975685-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704375975685-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:44965
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8010
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704375975685-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:53:35.866482 8010 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:53:35.867542 8010 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 17:53:35.877084 8010 server_base.cc:1034] running on GCE node
W20260929 17:53:35.876961 8015 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:53:35.877053 8016 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:53:35.876961 8018 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:53:35.877612 8010 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:53:35.878275 8010 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:53:35.879437 8010 hybrid_clock.cc:648] HybridClock initialized: now 1790704415879416 us; error 35 us; skew 500 ppm
Sep 29 17:53:35 dist-test-slave-53nh krb5kdc[7893](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704415, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:53:35.882035 8010 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260929 17:53:35.882915 8010 webserver.cc:460] Webserver started at http://127.0.98.1:38173/ using document root <none> and password file <none>
I20260929 17:53:35.883440 8010 fs_manager.cc:359] Metadata directory not provided
I20260929 17:53:35.883503 8010 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:53:35.883715 8010 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:53:35.885864 8010 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704375975685-392-0/minicluster-data/ts-0/data/instance:
uuid: "ece5590526f94058b07afe5bb610ccfa"
format_stamp: "Formatted at 2026-09-29 17:53:35 on dist-test-slave-53nh"
server_key: "6a67511548d59e89f0ca8832126ebea4"
server_key_iv: "b9443fb59eb74ada183dd873ea8aec05"
server_key_version: "encryptionkey@0"
I20260929 17:53:35.886390 8010 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704375975685-392-0/minicluster-data/ts-0/wal/instance:
uuid: "ece5590526f94058b07afe5bb610ccfa"
format_stamp: "Formatted at 2026-09-29 17:53:35 on dist-test-slave-53nh"
server_key: "6a67511548d59e89f0ca8832126ebea4"
server_key_iv: "b9443fb59eb74ada183dd873ea8aec05"
server_key_version: "encryptionkey@0"
I20260929 17:53:35.890024 8010 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 17:53:35.892398 8025 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:53:35.893679 8010 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 17:53:35.893787 8010 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704375975685-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704375975685-392-0/minicluster-data/ts-0/wal
uuid: "ece5590526f94058b07afe5bb610ccfa"
format_stamp: "Formatted at 2026-09-29 17:53:35 on dist-test-slave-53nh"
server_key: "6a67511548d59e89f0ca8832126ebea4"
server_key_iv: "b9443fb59eb74ada183dd873ea8aec05"
server_key_version: "encryptionkey@0"
I20260929 17:53:35.893877 8010 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704375975685-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704375975685-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704375975685-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:53:35.916281 8010 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:53:35.919238 8010 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:53:35.919423 8010 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:53:35.920833 8010 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:53:35.920882 8010 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:53:35.920933 8010 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:53:35.920987 8010 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 17:53:35 dist-test-slave-53nh krb5kdc[7893](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704415, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260929 17:53:35.931407 8010 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40515
I20260929 17:53:35.931468 8140 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40515 every 8 connection(s)
I20260929 17:53:35.932277 8010 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704375975685-392-0/minicluster-data/ts-0/data/info.pb
I20260929 17:53:35.934018 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskup3Soe/build/debug/bin/kudu as pid 8010
I20260929 17:53:35.934137 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704375975685-392-0/minicluster-data/ts-0/wal/instance
I20260929 17:53:35.934458 392 external_mini_cluster.cc:1428] Setting key 404d7b3f62ffb4a3dae0a2183844948e
I20260929 17:53:35.935225 7987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:35.921782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45449 (local address 127.0.98.62:44965)
0929 17:53:35.921889 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:35.921893 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:35.922925 (+ 1032us) server_negotiation.cc:407] Connection header received
0929 17:53:35.924005 (+ 1080us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:35.924008 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:35.924063 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:35.924128 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:53:35.925693 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:35.926511 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:35.928214 (+ 1703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:35.928461 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:35.928549 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:35.931676 (+ 3127us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:53:35.931680 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:53:35.931683 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:53:35.931713 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:53:35.933244 (+ 1531us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:53:35.933686 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:53:35.933688 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:53:35.933690 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:53:35.933760 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:53:35.934023 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:53:35.934026 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:53:35.934028 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:53:35.934254 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:53:35.934406 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0929 17:53:35.934888 (+ 482us) server_negotiation.cc:299] Negotiation successful
0929 17:53:35.935020 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 17:53:35.936017 8034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:35.922314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44965 (local address 127.0.98.1:45449)
0929 17:53:35.922778 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:35.922820 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 17:53:35.923766 (+ 946us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:53:35.924302 (+ 536us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:53:35.924312 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:53:35.924748 (+ 436us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:53:35.925457 (+ 709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:35.925475 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:35.926668 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:35.926672 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:35.928080 (+ 1408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:35.928091 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:35.928617 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:35.928622 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:35.928820 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:35.929582 (+ 762us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:53:35.929605 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:53:35.931472 (+ 1867us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:53:35.933347 (+ 1875us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:53:35.933351 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:53:35.933363 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:53:35.933606 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:53:35.933827 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:53:35.933829 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:53:35.933830 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:53:35.933945 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:53:35.934397 (+ 452us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:53:35.934402 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:53:35.934665 (+ 263us) client_negotiation.cc:769] Sending connection context
0929 17:53:35.934827 (+ 162us) client_negotiation.cc:240] Negotiation successful
0929 17:53:35.935137 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":121,"threads_started":1}
I20260929 17:53:35.937235 8141 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44965
I20260929 17:53:35.937368 8141 heartbeater.cc:464] Registering TS with master...
I20260929 17:53:35.937812 8141 heartbeater.cc:511] Master 127.0.98.62:44965 requested a full tablet report, sending...
I20260929 17:53:35.939234 7937 ts_manager.cc:194] Registered new tserver with Master: ece5590526f94058b07afe5bb610ccfa (127.0.98.1:40515)
I20260929 17:53:35.940549 7937 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:45449
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260929 17:53:35.946794 7987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:35.939408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35107 (local address 127.0.98.62:44965)
0929 17:53:35.939516 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:35.939520 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:35.939533 (+ 13us) server_negotiation.cc:407] Connection header received
0929 17:53:35.939598 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:35.939600 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:35.939642 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:35.939716 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:53:35.940186 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:35.940928 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:35.942099 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:35.942427 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:35.942537 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:35.943678 (+ 1141us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:53:35.943681 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:53:35.943684 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:53:35.943708 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:53:35.945314 (+ 1606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:53:35.945745 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:53:35.945748 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:53:35.945749 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:53:35.945826 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:53:35.946079 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:53:35.946081 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:53:35.946082 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:53:35.946235 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:53:35.946306 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0929 17:53:35.946572 (+ 266us) server_negotiation.cc:299] Negotiation successful
0929 17:53:35.946663 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 17:53:35.946812 8148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:35.939151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44965 (local address 127.0.98.1:35107)
0929 17:53:35.939423 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:35.939437 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 17:53:35.939517 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:53:35.939742 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:53:35.939745 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:53:35.939939 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:53:35.940058 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:35.940065 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:35.941076 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:35.941079 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:35.941980 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:35.941990 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:35.942554 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:35.942558 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:35.942645 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:35.943212 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:53:35.943223 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:53:35.943537 (+ 314us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:53:35.945417 (+ 1880us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:53:35.945420 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:53:35.945422 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:53:35.945655 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:53:35.945922 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:53:35.945925 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:53:35.945926 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:53:35.946010 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:53:35.946344 (+ 334us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:53:35.946347 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:53:35.946433 (+ 86us) client_negotiation.cc:769] Sending connection context
0929 17:53:35.946526 (+ 93us) client_negotiation.cc:240] Negotiation successful
0929 17:53:35.946639 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":96,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704375975685-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704375975685-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260929 17:53:35.961647 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskup3Soe/build/debug/bin/kudu
/tmp/dist-test-taskup3Soe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704375975685-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704375975685-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704375975685-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704375975685-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44965
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704375975685-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704375975685-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704375975685-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704375975685-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:53:36.077126 8150 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:53:36.077378 8150 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:53:36.077419 8150 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:53:36.081312 8150 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:53:36.081363 8150 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:53:36.081400 8150 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 17:53:36.081436 8150 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 17:53:36.086714 8150 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704375975685-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704375975685-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704375975685-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704375975685-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:44965
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8150
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704375975685-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:53:36.087682 8150 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:53:36.088668 8150 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:53:36.097785 8156 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:53:36.097793 8155 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:53:36.097788 8158 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:53:36.098037 8150 server_base.cc:1034] running on GCE node
I20260929 17:53:36.098508 8150 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:53:36.099128 8150 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:53:36.100277 8150 hybrid_clock.cc:648] HybridClock initialized: now 1790704416100251 us; error 34 us; skew 500 ppm
Sep 29 17:53:36 dist-test-slave-53nh krb5kdc[7893](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704416, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:53:36.102934 8150 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260929 17:53:36.103857 8150 webserver.cc:460] Webserver started at http://127.0.98.2:41683/ using document root <none> and password file <none>
I20260929 17:53:36.104373 8150 fs_manager.cc:359] Metadata directory not provided
I20260929 17:53:36.104434 8150 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:53:36.104627 8150 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:53:36.106657 8150 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704375975685-392-0/minicluster-data/ts-1/data/instance:
uuid: "4e821d34d1844842be7365e97204614e"
format_stamp: "Formatted at 2026-09-29 17:53:36 on dist-test-slave-53nh"
server_key: "52ba29d2e2235b73294cb4138a71b677"
server_key_iv: "cfb6d8e7568d37ac694a80174e5b3c26"
server_key_version: "encryptionkey@0"
I20260929 17:53:36.107189 8150 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704375975685-392-0/minicluster-data/ts-1/wal/instance:
uuid: "4e821d34d1844842be7365e97204614e"
format_stamp: "Formatted at 2026-09-29 17:53:36 on dist-test-slave-53nh"
server_key: "52ba29d2e2235b73294cb4138a71b677"
server_key_iv: "cfb6d8e7568d37ac694a80174e5b3c26"
server_key_version: "encryptionkey@0"
I20260929 17:53:36.110713 8150 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260929 17:53:36.113269 8165 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:53:36.114528 8150 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 17:53:36.114636 8150 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704375975685-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704375975685-392-0/minicluster-data/ts-1/wal
uuid: "4e821d34d1844842be7365e97204614e"
format_stamp: "Formatted at 2026-09-29 17:53:36 on dist-test-slave-53nh"
server_key: "52ba29d2e2235b73294cb4138a71b677"
server_key_iv: "cfb6d8e7568d37ac694a80174e5b3c26"
server_key_version: "encryptionkey@0"
I20260929 17:53:36.114729 8150 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704375975685-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704375975685-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704375975685-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:53:36.134049 8150 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:53:36.137239 8150 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:53:36.137430 8150 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:53:36.138880 8150 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:53:36.138931 8150 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:53:36.138983 8150 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:53:36.139012 8150 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 17:53:36 dist-test-slave-53nh krb5kdc[7893](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704416, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260929 17:53:36.148730 8150 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43829
I20260929 17:53:36.148841 8280 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43829 every 8 connection(s)
I20260929 17:53:36.149664 8150 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704375975685-392-0/minicluster-data/ts-1/data/info.pb
I20260929 17:53:36.152661 7987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:36.139651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36651 (local address 127.0.98.62:44965)
0929 17:53:36.139791 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:36.139796 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:36.140643 (+ 847us) server_negotiation.cc:407] Connection header received
0929 17:53:36.141873 (+ 1230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:36.141878 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:36.141939 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:36.142037 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:53:36.143187 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:36.144004 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:36.145743 (+ 1739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:36.146033 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:36.146125 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:36.149175 (+ 3050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:53:36.149179 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:53:36.149181 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:53:36.149206 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:53:36.150669 (+ 1463us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:53:36.151218 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:53:36.151223 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:53:36.151224 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:53:36.151310 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:53:36.151621 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:53:36.151625 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:53:36.151627 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:53:36.151809 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:53:36.151892 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0929 17:53:36.152374 (+ 482us) server_negotiation.cc:299] Negotiation successful
0929 17:53:36.152503 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 17:53:36.153472 8174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:36.140047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44965 (local address 127.0.98.2:36651)
0929 17:53:36.140493 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:36.140543 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0929 17:53:36.141582 (+ 1039us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:53:36.142218 (+ 636us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:53:36.142228 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:53:36.142561 (+ 333us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:53:36.143052 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:36.143065 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:36.144143 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:36.144147 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:36.145607 (+ 1460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:36.145619 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:36.146265 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:36.146276 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:36.146513 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:36.147213 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:53:36.147236 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:53:36.148986 (+ 1750us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:53:36.150785 (+ 1799us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:53:36.150790 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:53:36.150805 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:53:36.151112 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:53:36.151390 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:53:36.151393 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:53:36.151395 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:53:36.151528 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:53:36.151903 (+ 375us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:53:36.151910 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:53:36.152142 (+ 232us) client_negotiation.cc:769] Sending connection context
0929 17:53:36.152298 (+ 156us) client_negotiation.cc:240] Negotiation successful
0929 17:53:36.152616 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":99,"threads_started":1}
I20260929 17:53:36.154667 8281 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44965
I20260929 17:53:36.154807 8281 heartbeater.cc:464] Registering TS with master...
I20260929 17:53:36.155342 8281 heartbeater.cc:511] Master 127.0.98.62:44965 requested a full tablet report, sending...
I20260929 17:53:36.156364 7938 ts_manager.cc:194] Registered new tserver with Master: 4e821d34d1844842be7365e97204614e (127.0.98.2:43829)
I20260929 17:53:36.157088 7938 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:36651
I20260929 17:53:36.157858 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskup3Soe/build/debug/bin/kudu as pid 8150
I20260929 17:53:36.157977 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704375975685-392-0/minicluster-data/ts-1/wal/instance
I20260929 17:53:36.158246 392 external_mini_cluster.cc:1428] Setting key 789003f8c809715903669e39a05b9c5d
I20260929 17:53:36.163739 7987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:36.156906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55349 (local address 127.0.98.62:44965)
0929 17:53:36.157018 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:36.157022 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:36.157254 (+ 232us) server_negotiation.cc:407] Connection header received
0929 17:53:36.157373 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:36.157376 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:36.157417 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:36.157478 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:53:36.158021 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:36.158731 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:36.159672 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:36.159969 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:36.160069 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:36.161066 (+ 997us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:53:36.161070 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:53:36.161073 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:53:36.161095 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:53:36.162295 (+ 1200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:53:36.162665 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:53:36.162669 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:53:36.162670 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:53:36.162750 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:53:36.162981 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:53:36.162984 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:53:36.162986 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:53:36.163179 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:53:36.163263 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0929 17:53:36.163491 (+ 228us) server_negotiation.cc:299] Negotiation successful
0929 17:53:36.163589 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260929 17:53:36.163745 8287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:36.156835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44965 (local address 127.0.98.2:55349)
0929 17:53:36.157180 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:36.157193 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 17:53:36.157289 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:53:36.157504 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:53:36.157507 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:53:36.157740 (+ 233us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:53:36.157880 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:36.157888 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:36.158941 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:36.158944 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:36.159563 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:36.159572 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:36.160064 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:36.160066 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:36.160127 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:36.160608 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:53:36.160618 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:53:36.160916 (+ 298us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:53:36.162389 (+ 1473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:53:36.162392 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:53:36.162394 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:53:36.162583 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:53:36.162826 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:53:36.162829 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:53:36.162832 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:53:36.162907 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:53:36.163264 (+ 357us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:53:36.163267 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:53:36.163352 (+ 85us) client_negotiation.cc:769] Sending connection context
0929 17:53:36.163432 (+ 80us) client_negotiation.cc:240] Negotiation successful
0929 17:53:36.163565 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":95,"threads_started":1}
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704375975685-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704375975685-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260929 17:53:36.183149 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskup3Soe/build/debug/bin/kudu
/tmp/dist-test-taskup3Soe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704375975685-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704375975685-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704375975685-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704375975685-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44965
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704375975685-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704375975685-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704375975685-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704375975685-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:53:36.290268 8290 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:53:36.290500 8290 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:53:36.290540 8290 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:53:36.294448 8290 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:53:36.294502 8290 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:53:36.294538 8290 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 17:53:36.294569 8290 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 17:53:36.299697 8290 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704375975685-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704375975685-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704375975685-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704375975685-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:44965
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8290
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704375975685-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:53:36.300647 8290 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:53:36.301630 8290 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:53:36.310747 8295 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:53:36.310752 8296 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:53:36.310760 8298 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:53:36.310866 8290 server_base.cc:1034] running on GCE node
I20260929 17:53:36.311353 8290 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:53:36.311980 8290 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:53:36.313140 8290 hybrid_clock.cc:648] HybridClock initialized: now 1790704416313123 us; error 33 us; skew 500 ppm
Sep 29 17:53:36 dist-test-slave-53nh krb5kdc[7893](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704416, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:53:36.315856 8290 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260929 17:53:36.316695 8290 webserver.cc:460] Webserver started at http://127.0.98.3:38633/ using document root <none> and password file <none>
I20260929 17:53:36.317257 8290 fs_manager.cc:359] Metadata directory not provided
I20260929 17:53:36.317327 8290 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:53:36.317519 8290 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:53:36.319540 8290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704375975685-392-0/minicluster-data/ts-2/data/instance:
uuid: "c8be608a8c8b43e6a7063022a8a7da36"
format_stamp: "Formatted at 2026-09-29 17:53:36 on dist-test-slave-53nh"
server_key: "0cd816bae03dd75052097f9215f783af"
server_key_iv: "18619e40e5d3ad1c68e381c06d2cd9ef"
server_key_version: "encryptionkey@0"
I20260929 17:53:36.320070 8290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704375975685-392-0/minicluster-data/ts-2/wal/instance:
uuid: "c8be608a8c8b43e6a7063022a8a7da36"
format_stamp: "Formatted at 2026-09-29 17:53:36 on dist-test-slave-53nh"
server_key: "0cd816bae03dd75052097f9215f783af"
server_key_iv: "18619e40e5d3ad1c68e381c06d2cd9ef"
server_key_version: "encryptionkey@0"
I20260929 17:53:36.323590 8290 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260929 17:53:36.326109 8305 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:53:36.327338 8290 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 17:53:36.327451 8290 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704375975685-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704375975685-392-0/minicluster-data/ts-2/wal
uuid: "c8be608a8c8b43e6a7063022a8a7da36"
format_stamp: "Formatted at 2026-09-29 17:53:36 on dist-test-slave-53nh"
server_key: "0cd816bae03dd75052097f9215f783af"
server_key_iv: "18619e40e5d3ad1c68e381c06d2cd9ef"
server_key_version: "encryptionkey@0"
I20260929 17:53:36.327544 8290 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704375975685-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704375975685-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704375975685-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:53:36.346443 8290 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:53:36.349684 8290 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:53:36.349872 8290 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:53:36.351286 8290 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:53:36.351334 8290 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:53:36.351387 8290 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:53:36.351414 8290 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 17:53:36 dist-test-slave-53nh krb5kdc[7893](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704416, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260929 17:53:36.361436 8290 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:32913
I20260929 17:53:36.361481 8420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:32913 every 8 connection(s)
I20260929 17:53:36.362469 8290 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704375975685-392-0/minicluster-data/ts-2/data/info.pb
I20260929 17:53:36.364356 7987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:36.352164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57183 (local address 127.0.98.62:44965)
0929 17:53:36.352275 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:36.352279 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:36.353394 (+ 1115us) server_negotiation.cc:407] Connection header received
0929 17:53:36.354424 (+ 1030us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:36.354427 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:36.354485 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:36.354568 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:53:36.356099 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:36.356916 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:36.358195 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:36.358417 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:36.358504 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:36.360777 (+ 2273us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:53:36.360781 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:53:36.360784 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:53:36.360810 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:53:36.362366 (+ 1556us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:53:36.362895 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:53:36.362899 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:53:36.362900 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:53:36.362985 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:53:36.363327 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:53:36.363330 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:53:36.363332 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:53:36.363498 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:53:36.363581 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0929 17:53:36.364044 (+ 463us) server_negotiation.cc:299] Negotiation successful
0929 17:53:36.364207 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260929 17:53:36.365077 8314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:36.352724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44965 (local address 127.0.98.3:57183)
0929 17:53:36.353253 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:36.353295 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 17:53:36.354193 (+ 898us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:53:36.354786 (+ 593us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:53:36.354796 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:53:36.355247 (+ 451us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:53:36.355944 (+ 697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:36.355964 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:36.357052 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:36.357056 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:36.358077 (+ 1021us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:36.358086 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:36.358547 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:36.358551 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:36.358721 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:36.359232 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:53:36.359247 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:53:36.360617 (+ 1370us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:53:36.362473 (+ 1856us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:53:36.362477 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:53:36.362487 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:53:36.362777 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:53:36.363081 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:53:36.363084 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:53:36.363085 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:53:36.363238 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:53:36.363591 (+ 353us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:53:36.363605 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:53:36.363833 (+ 228us) client_negotiation.cc:769] Sending connection context
0929 17:53:36.363988 (+ 155us) client_negotiation.cc:240] Negotiation successful
0929 17:53:36.364275 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":123,"threads_started":1}
I20260929 17:53:36.366196 8421 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44965
I20260929 17:53:36.366329 8421 heartbeater.cc:464] Registering TS with master...
I20260929 17:53:36.366806 8421 heartbeater.cc:511] Master 127.0.98.62:44965 requested a full tablet report, sending...
I20260929 17:53:36.367726 7937 ts_manager.cc:194] Registered new tserver with Master: c8be608a8c8b43e6a7063022a8a7da36 (127.0.98.3:32913)
I20260929 17:53:36.368425 7937 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:57183
I20260929 17:53:36.369238 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskup3Soe/build/debug/bin/kudu as pid 8290
I20260929 17:53:36.369350 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704375975685-392-0/minicluster-data/ts-2/wal/instance
I20260929 17:53:36.369601 392 external_mini_cluster.cc:1428] Setting key 26f23c90ca17fd7a782355b83fdda985
I20260929 17:53:36.371332 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 17:53:36.374858 8427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:36.367947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44965 (local address 127.0.98.3:53751)
0929 17:53:36.368197 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:36.368212 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 17:53:36.368301 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:53:36.368486 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:53:36.368488 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:53:36.368670 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:53:36.368776 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:36.368782 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:36.369768 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:36.369772 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:36.370542 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:36.370550 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:36.370950 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:36.370953 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:36.371035 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:36.371495 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:53:36.371508 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:53:36.371890 (+ 382us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:53:36.373595 (+ 1705us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:53:36.373597 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:53:36.373599 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:53:36.373779 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:53:36.374023 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:53:36.374025 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:53:36.374026 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:53:36.374090 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:53:36.374401 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:53:36.374403 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:53:36.374472 (+ 69us) client_negotiation.cc:769] Sending connection context
0929 17:53:36.374550 (+ 78us) client_negotiation.cc:240] Negotiation successful
0929 17:53:36.374673 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":84,"threads_started":1}
I20260929 17:53:36.374866 7987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:36.368039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53751 (local address 127.0.98.62:44965)
0929 17:53:36.368144 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:36.368148 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:36.368272 (+ 124us) server_negotiation.cc:407] Connection header received
0929 17:53:36.368365 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:36.368367 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:36.368399 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:36.368455 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:53:36.368878 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:36.369622 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:36.370647 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:36.370858 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:36.370939 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:36.372029 (+ 1090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:53:36.372032 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:53:36.372033 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:53:36.372087 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:53:36.373497 (+ 1410us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:53:36.373861 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:53:36.373864 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:53:36.373865 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:53:36.373944 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:53:36.374156 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:53:36.374158 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:53:36.374160 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:53:36.374334 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:53:36.374406 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0929 17:53:36.374628 (+ 222us) server_negotiation.cc:299] Negotiation successful
0929 17:53:36.374729 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
Sep 29 17:53:36 dist-test-slave-53nh krb5kdc[7893](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704416, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 17:53:36 dist-test-slave-53nh krb5kdc[7893](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 17:53:36 dist-test-slave-53nh krb5kdc[7893](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260929 17:53:36.393627 7987 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:53:36.388092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51920 (local address 127.0.98.62:44965)
0929 17:53:36.388205 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:36.388208 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:36.388219 (+ 11us) server_negotiation.cc:407] Connection header received
0929 17:53:36.388252 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:36.388254 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:36.388291 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:36.388355 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:53:36.389037 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:36.389545 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:36.390362 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:36.390568 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:36.390651 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:36.393483 (+ 2832us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51920: BlockingRecv error: recv got EOF from 127.0.0.1:51920 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Sep 29 17:53:36 dist-test-slave-53nh krb5kdc[7893](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704416, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260929 17:53:36.402220 7987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:36.394943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51932 (local address 127.0.98.62:44965)
0929 17:53:36.395052 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:36.395056 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:36.395158 (+ 102us) server_negotiation.cc:407] Connection header received
0929 17:53:36.395272 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:36.395275 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:36.395319 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:36.395380 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:53:36.395931 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:36.396437 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:36.397227 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:36.397447 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:36.397533 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:36.399595 (+ 2062us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:53:36.399598 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:53:36.399600 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:53:36.399622 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:53:36.400740 (+ 1118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:53:36.401152 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:53:36.401156 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:53:36.401158 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:53:36.401232 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:53:36.401468 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:53:36.401471 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:53:36.401473 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:53:36.401641 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:53:36.401770 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0929 17:53:36.401968 (+ 198us) server_negotiation.cc:299] Negotiation successful
0929 17:53:36.402071 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260929 17:53:36.404863 7937 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51932:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 17:53:36.407828 7937 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 17:53:36.422577 8448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:36.417221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45988 (local address 127.0.98.2:43829)
0929 17:53:36.417775 (+ 554us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:36.417787 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:36.417975 (+ 188us) server_negotiation.cc:407] Connection header received
0929 17:53:36.418050 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:36.418054 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:36.418257 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:36.418382 (+ 125us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:53:36.418724 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:36.419803 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:36.420992 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:36.421820 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:36.421930 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:36.422108 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0929 17:53:36.422321 (+ 213us) server_negotiation.cc:299] Negotiation successful
0929 17:53:36.422419 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":416,"thread_start_us":93,"threads_started":1}
I20260929 17:53:36.422593 8443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:36.416268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:32913 (local address 127.0.0.1:43864)
0929 17:53:36.416696 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:36.416743 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 17:53:36.416859 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:53:36.418102 (+ 1243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:53:36.418108 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:53:36.418127 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:53:36.418256 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:36.418271 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:36.419666 (+ 1395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:36.419669 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:36.421081 (+ 1412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:36.421092 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:36.422024 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:36.422027 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:36.422168 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:36.422185 (+ 17us) client_negotiation.cc:769] Sending connection context
0929 17:53:36.422280 (+ 95us) client_negotiation.cc:240] Negotiation successful
0929 17:53:36.422439 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"mutex_wait_us":175,"thread_start_us":86,"threads_started":1}
I20260929 17:53:36.422593 8445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:36.416984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43829 (local address 127.0.0.1:45988)
0929 17:53:36.417886 (+ 902us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:36.417896 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 17:53:36.417967 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:53:36.418526 (+ 559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:53:36.418530 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:53:36.418546 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:53:36.418618 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:36.418624 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:36.419923 (+ 1299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:36.419927 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:36.420869 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:36.420878 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:36.421927 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:36.421929 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:36.422117 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:36.422168 (+ 51us) client_negotiation.cc:769] Sending connection context
0929 17:53:36.422280 (+ 112us) client_negotiation.cc:240] Negotiation successful
0929 17:53:36.422425 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":849,"mutex_wait_us":76,"thread_start_us":87,"threads_started":1}
I20260929 17:53:36.423086 8444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:36.416412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43864 (local address 127.0.98.3:32913)
0929 17:53:36.417329 (+ 917us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:36.417342 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:36.417366 (+ 24us) server_negotiation.cc:407] Connection header received
0929 17:53:36.417442 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:36.417448 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:36.417668 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:36.418107 (+ 439us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:53:36.418374 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:36.419543 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:36.421204 (+ 1661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:36.421916 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:36.422007 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:36.422133 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0929 17:53:36.422829 (+ 696us) server_negotiation.cc:299] Negotiation successful
0929 17:53:36.422951 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":814,"thread_start_us":122,"threads_started":1}
I20260929 17:53:36.423722 8446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:36.416844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40515 (local address 127.0.0.1:32840)
0929 17:53:36.417469 (+ 625us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:36.417482 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 17:53:36.417570 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:53:36.418341 (+ 771us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:53:36.418344 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:53:36.418361 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:53:36.418468 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:36.418474 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:36.419564 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:36.419567 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:36.421081 (+ 1514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:36.421092 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:36.423355 (+ 2263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:36.423359 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:36.423452 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:36.423473 (+ 21us) client_negotiation.cc:769] Sending connection context
0929 17:53:36.423537 (+ 64us) client_negotiation.cc:240] Negotiation successful
0929 17:53:36.423600 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":566,"mutex_wait_us":60,"thread_start_us":98,"threads_started":1}
I20260929 17:53:36.424181 8447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:36.417046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32840 (local address 127.0.98.1:40515)
0929 17:53:36.417783 (+ 737us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:36.417796 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:36.417818 (+ 22us) server_negotiation.cc:407] Connection header received
0929 17:53:36.417879 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:36.417882 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:36.418052 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:36.418195 (+ 143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:53:36.418607 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:36.419454 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:36.422190 (+ 2736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:36.423022 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:36.423125 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:36.423295 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0929 17:53:36.423954 (+ 659us) server_negotiation.cc:299] Negotiation successful
0929 17:53:36.424047 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":648,"thread_start_us":61,"threads_started":1}
I20260929 17:53:36.425920 8215 tablet_service.cc:1467] Processing CreateTablet for tablet e71e7b3f334b4447a6ea39b42ca38a5b (DEFAULT_TABLE table=test-table [id=4f2c328f95bd4e87bc0c6ef87d8bb1f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 17:53:36.425947 8355 tablet_service.cc:1467] Processing CreateTablet for tablet e71e7b3f334b4447a6ea39b42ca38a5b (DEFAULT_TABLE table=test-table [id=4f2c328f95bd4e87bc0c6ef87d8bb1f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 17:53:36.426862 8215 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e71e7b3f334b4447a6ea39b42ca38a5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:53:36.427000 8355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e71e7b3f334b4447a6ea39b42ca38a5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:53:36.426770 8075 tablet_service.cc:1467] Processing CreateTablet for tablet e71e7b3f334b4447a6ea39b42ca38a5b (DEFAULT_TABLE table=test-table [id=4f2c328f95bd4e87bc0c6ef87d8bb1f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 17:53:36.427640 8075 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e71e7b3f334b4447a6ea39b42ca38a5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:53:36.433311 8449 tablet_bootstrap.cc:492] T e71e7b3f334b4447a6ea39b42ca38a5b P 4e821d34d1844842be7365e97204614e: Bootstrap starting.
I20260929 17:53:36.434614 8450 tablet_bootstrap.cc:492] T e71e7b3f334b4447a6ea39b42ca38a5b P c8be608a8c8b43e6a7063022a8a7da36: Bootstrap starting.
I20260929 17:53:36.435509 8451 tablet_bootstrap.cc:492] T e71e7b3f334b4447a6ea39b42ca38a5b P ece5590526f94058b07afe5bb610ccfa: Bootstrap starting.
I20260929 17:53:36.436695 8450 tablet_bootstrap.cc:654] T e71e7b3f334b4447a6ea39b42ca38a5b P c8be608a8c8b43e6a7063022a8a7da36: Neither blocks nor log segments found. Creating new log.
I20260929 17:53:36.436695 8449 tablet_bootstrap.cc:654] T e71e7b3f334b4447a6ea39b42ca38a5b P 4e821d34d1844842be7365e97204614e: Neither blocks nor log segments found. Creating new log.
I20260929 17:53:36.437315 8451 tablet_bootstrap.cc:654] T e71e7b3f334b4447a6ea39b42ca38a5b P ece5590526f94058b07afe5bb610ccfa: Neither blocks nor log segments found. Creating new log.
I20260929 17:53:36.437613 8449 log.cc:824] T e71e7b3f334b4447a6ea39b42ca38a5b P 4e821d34d1844842be7365e97204614e: Log is configured to *not* fsync() on all Append() calls
I20260929 17:53:36.437613 8450 log.cc:824] T e71e7b3f334b4447a6ea39b42ca38a5b P c8be608a8c8b43e6a7063022a8a7da36: Log is configured to *not* fsync() on all Append() calls
I20260929 17:53:36.437997 8451 log.cc:824] T e71e7b3f334b4447a6ea39b42ca38a5b P ece5590526f94058b07afe5bb610ccfa: Log is configured to *not* fsync() on all Append() calls
I20260929 17:53:36.439657 8450 tablet_bootstrap.cc:492] T e71e7b3f334b4447a6ea39b42ca38a5b P c8be608a8c8b43e6a7063022a8a7da36: No bootstrap required, opened a new log
I20260929 17:53:36.439673 8449 tablet_bootstrap.cc:492] T e71e7b3f334b4447a6ea39b42ca38a5b P 4e821d34d1844842be7365e97204614e: No bootstrap required, opened a new log
I20260929 17:53:36.439853 8450 ts_tablet_manager.cc:1397] T e71e7b3f334b4447a6ea39b42ca38a5b P c8be608a8c8b43e6a7063022a8a7da36: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260929 17:53:36.439853 8449 ts_tablet_manager.cc:1397] T e71e7b3f334b4447a6ea39b42ca38a5b P 4e821d34d1844842be7365e97204614e: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260929 17:53:36.441449 8451 tablet_bootstrap.cc:492] T e71e7b3f334b4447a6ea39b42ca38a5b P ece5590526f94058b07afe5bb610ccfa: No bootstrap required, opened a new log
I20260929 17:53:36.441659 8451 ts_tablet_manager.cc:1397] T e71e7b3f334b4447a6ea39b42ca38a5b P ece5590526f94058b07afe5bb610ccfa: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 17:53:36.443874 8449 raft_consensus.cc:348] T e71e7b3f334b4447a6ea39b42ca38a5b P 4e821d34d1844842be7365e97204614e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8be608a8c8b43e6a7063022a8a7da36" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32913 } } peers { permanent_uuid: "4e821d34d1844842be7365e97204614e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43829 } } peers { permanent_uuid: "ece5590526f94058b07afe5bb610ccfa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40515 } }
I20260929 17:53:36.443874 8450 raft_consensus.cc:348] T e71e7b3f334b4447a6ea39b42ca38a5b P c8be608a8c8b43e6a7063022a8a7da36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8be608a8c8b43e6a7063022a8a7da36" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32913 } } peers { permanent_uuid: "4e821d34d1844842be7365e97204614e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43829 } } peers { permanent_uuid: "ece5590526f94058b07afe5bb610ccfa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40515 } }
I20260929 17:53:36.444175 8449 raft_consensus.cc:374] T e71e7b3f334b4447a6ea39b42ca38a5b P 4e821d34d1844842be7365e97204614e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:53:36.444175 8450 raft_consensus.cc:374] T e71e7b3f334b4447a6ea39b42ca38a5b P c8be608a8c8b43e6a7063022a8a7da36 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:53:36.444244 8450 raft_consensus.cc:729] T e71e7b3f334b4447a6ea39b42ca38a5b P c8be608a8c8b43e6a7063022a8a7da36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8be608a8c8b43e6a7063022a8a7da36, State: Initialized, Role: FOLLOWER
I20260929 17:53:36.444280 8449 raft_consensus.cc:729] T e71e7b3f334b4447a6ea39b42ca38a5b P 4e821d34d1844842be7365e97204614e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e821d34d1844842be7365e97204614e, State: Initialized, Role: FOLLOWER
I20260929 17:53:36.444453 8451 raft_consensus.cc:348] T e71e7b3f334b4447a6ea39b42ca38a5b P ece5590526f94058b07afe5bb610ccfa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8be608a8c8b43e6a7063022a8a7da36" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32913 } } peers { permanent_uuid: "4e821d34d1844842be7365e97204614e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43829 } } peers { permanent_uuid: "ece5590526f94058b07afe5bb610ccfa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40515 } }
I20260929 17:53:36.444713 8451 raft_consensus.cc:374] T e71e7b3f334b4447a6ea39b42ca38a5b P ece5590526f94058b07afe5bb610ccfa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:53:36.444794 8451 raft_consensus.cc:729] T e71e7b3f334b4447a6ea39b42ca38a5b P ece5590526f94058b07afe5bb610ccfa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ece5590526f94058b07afe5bb610ccfa, State: Initialized, Role: FOLLOWER
I20260929 17:53:36.444751 8450 consensus_queue.cc:260] T e71e7b3f334b4447a6ea39b42ca38a5b P c8be608a8c8b43e6a7063022a8a7da36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8be608a8c8b43e6a7063022a8a7da36" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32913 } } peers { permanent_uuid: "4e821d34d1844842be7365e97204614e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43829 } } peers { permanent_uuid: "ece5590526f94058b07afe5bb610ccfa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40515 } }
I20260929 17:53:36.444751 8449 consensus_queue.cc:260] T e71e7b3f334b4447a6ea39b42ca38a5b P 4e821d34d1844842be7365e97204614e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8be608a8c8b43e6a7063022a8a7da36" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32913 } } peers { permanent_uuid: "4e821d34d1844842be7365e97204614e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43829 } } peers { permanent_uuid: "ece5590526f94058b07afe5bb610ccfa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40515 } }
I20260929 17:53:36.445219 8451 consensus_queue.cc:260] T e71e7b3f334b4447a6ea39b42ca38a5b P ece5590526f94058b07afe5bb610ccfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8be608a8c8b43e6a7063022a8a7da36" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32913 } } peers { permanent_uuid: "4e821d34d1844842be7365e97204614e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43829 } } peers { permanent_uuid: "ece5590526f94058b07afe5bb610ccfa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40515 } }
I20260929 17:53:36.445474 8421 heartbeater.cc:503] Master 127.0.98.62:44965 was elected leader, sending a full tablet report...
I20260929 17:53:36.445698 8281 heartbeater.cc:503] Master 127.0.98.62:44965 was elected leader, sending a full tablet report...
I20260929 17:53:36.445874 8449 ts_tablet_manager.cc:1428] T e71e7b3f334b4447a6ea39b42ca38a5b P 4e821d34d1844842be7365e97204614e: Time spent starting tablet: real 0.006s user 0.006s sys 0.001s
I20260929 17:53:36.446653 8450 ts_tablet_manager.cc:1428] T e71e7b3f334b4447a6ea39b42ca38a5b P c8be608a8c8b43e6a7063022a8a7da36: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260929 17:53:36.447253 8451 ts_tablet_manager.cc:1428] T e71e7b3f334b4447a6ea39b42ca38a5b P ece5590526f94058b07afe5bb610ccfa: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260929 17:53:36.447295 8141 heartbeater.cc:503] Master 127.0.98.62:44965 was elected leader, sending a full tablet report...
I20260929 17:53:36.481050 8455 raft_consensus.cc:482] T e71e7b3f334b4447a6ea39b42ca38a5b P c8be608a8c8b43e6a7063022a8a7da36 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 17:53:36.481249 8455 raft_consensus.cc:504] T e71e7b3f334b4447a6ea39b42ca38a5b P c8be608a8c8b43e6a7063022a8a7da36 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8be608a8c8b43e6a7063022a8a7da36" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32913 } } peers { permanent_uuid: "4e821d34d1844842be7365e97204614e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43829 } } peers { permanent_uuid: "ece5590526f94058b07afe5bb610ccfa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40515 } }
I20260929 17:53:36.482404 8455 leader_election.cc:288] T e71e7b3f334b4447a6ea39b42ca38a5b P c8be608a8c8b43e6a7063022a8a7da36 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4e821d34d1844842be7365e97204614e (127.0.98.2:43829), ece5590526f94058b07afe5bb610ccfa (127.0.98.1:40515)
I20260929 17:53:36.486595 8314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:36.482628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40515 (local address 127.0.98.3:42361)
0929 17:53:36.482734 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:36.482748 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 17:53:36.482835 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:53:36.483105 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:53:36.483108 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:53:36.483128 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:53:36.483255 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:36.483262 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:36.484310 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:36.484313 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:36.485641 (+ 1328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:36.485651 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:36.486280 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:36.486283 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:36.486360 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:36.486380 (+ 20us) client_negotiation.cc:769] Sending connection context
0929 17:53:36.486433 (+ 53us) client_negotiation.cc:240] Negotiation successful
0929 17:53:36.486492 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31,"mutex_wait_us":3}
I20260929 17:53:36.486657 8447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:36.482769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42361 (local address 127.0.98.1:40515)
0929 17:53:36.482899 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:36.482903 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:36.482915 (+ 12us) server_negotiation.cc:407] Connection header received
0929 17:53:36.482963 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:36.482965 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:36.483005 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:36.483078 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:53:36.483364 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:36.484190 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:36.485769 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:36.486170 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:36.486270 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:36.486335 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0929 17:53:36.486466 (+ 131us) server_negotiation.cc:299] Negotiation successful
0929 17:53:36.486534 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 17:53:36.487278 8458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:36.482631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43829 (local address 127.0.98.3:49557)
0929 17:53:36.483260 (+ 629us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:36.483273 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 17:53:36.483353 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:53:36.483566 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:53:36.483568 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:53:36.483580 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:53:36.483694 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:36.483700 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:36.484566 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:36.484569 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:36.486061 (+ 1492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:36.486069 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:36.486913 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:36.486917 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:36.487000 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:36.487016 (+ 16us) client_negotiation.cc:769] Sending connection context
0929 17:53:36.487080 (+ 64us) client_negotiation.cc:240] Negotiation successful
0929 17:53:36.487141 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":573,"mutex_wait_us":263,"thread_start_us":123,"threads_started":1}
I20260929 17:53:36.487286 8448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:36.482735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49557 (local address 127.0.98.2:43829)
0929 17:53:36.482860 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:36.482864 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:36.483352 (+ 488us) server_negotiation.cc:407] Connection header received
0929 17:53:36.483426 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:36.483428 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:36.483477 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:36.483544 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:53:36.483847 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:36.484484 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:36.486184 (+ 1700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:36.486729 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:36.486827 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:36.486885 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0929 17:53:36.487103 (+ 218us) server_negotiation.cc:299] Negotiation successful
0929 17:53:36.487167 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 17:53:36.487284 8095 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e71e7b3f334b4447a6ea39b42ca38a5b" candidate_uuid: "c8be608a8c8b43e6a7063022a8a7da36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ece5590526f94058b07afe5bb610ccfa" is_pre_election: true
I20260929 17:53:36.487596 8095 raft_consensus.cc:2454] T e71e7b3f334b4447a6ea39b42ca38a5b P ece5590526f94058b07afe5bb610ccfa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c8be608a8c8b43e6a7063022a8a7da36 in term 0.
I20260929 17:53:36.487805 8235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e71e7b3f334b4447a6ea39b42ca38a5b" candidate_uuid: "c8be608a8c8b43e6a7063022a8a7da36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e821d34d1844842be7365e97204614e" is_pre_election: true
I20260929 17:53:36.488059 8235 raft_consensus.cc:2454] T e71e7b3f334b4447a6ea39b42ca38a5b P 4e821d34d1844842be7365e97204614e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c8be608a8c8b43e6a7063022a8a7da36 in term 0.
I20260929 17:53:36.488049 8306 leader_election.cc:302] T e71e7b3f334b4447a6ea39b42ca38a5b P c8be608a8c8b43e6a7063022a8a7da36 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c8be608a8c8b43e6a7063022a8a7da36, ece5590526f94058b07afe5bb610ccfa; no voters:
I20260929 17:53:36.488307 8455 raft_consensus.cc:2781] T e71e7b3f334b4447a6ea39b42ca38a5b P c8be608a8c8b43e6a7063022a8a7da36 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 17:53:36.488386 8455 raft_consensus.cc:482] T e71e7b3f334b4447a6ea39b42ca38a5b P c8be608a8c8b43e6a7063022a8a7da36 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 17:53:36.488438 8455 raft_consensus.cc:3037] T e71e7b3f334b4447a6ea39b42ca38a5b P c8be608a8c8b43e6a7063022a8a7da36 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:53:36.489802 8455 raft_consensus.cc:504] T e71e7b3f334b4447a6ea39b42ca38a5b P c8be608a8c8b43e6a7063022a8a7da36 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8be608a8c8b43e6a7063022a8a7da36" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32913 } } peers { permanent_uuid: "4e821d34d1844842be7365e97204614e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43829 } } peers { permanent_uuid: "ece5590526f94058b07afe5bb610ccfa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40515 } }
I20260929 17:53:36.490172 8455 leader_election.cc:288] T e71e7b3f334b4447a6ea39b42ca38a5b P c8be608a8c8b43e6a7063022a8a7da36 [CANDIDATE]: Term 1 election: Requested vote from peers 4e821d34d1844842be7365e97204614e (127.0.98.2:43829), ece5590526f94058b07afe5bb610ccfa (127.0.98.1:40515)
I20260929 17:53:36.490456 8235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e71e7b3f334b4447a6ea39b42ca38a5b" candidate_uuid: "c8be608a8c8b43e6a7063022a8a7da36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e821d34d1844842be7365e97204614e"
I20260929 17:53:36.490582 8235 raft_consensus.cc:3037] T e71e7b3f334b4447a6ea39b42ca38a5b P 4e821d34d1844842be7365e97204614e [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:53:36.491830 8235 raft_consensus.cc:2454] T e71e7b3f334b4447a6ea39b42ca38a5b P 4e821d34d1844842be7365e97204614e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8be608a8c8b43e6a7063022a8a7da36 in term 1.
I20260929 17:53:36.492146 8309 leader_election.cc:302] T e71e7b3f334b4447a6ea39b42ca38a5b P c8be608a8c8b43e6a7063022a8a7da36 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e821d34d1844842be7365e97204614e, c8be608a8c8b43e6a7063022a8a7da36; no voters:
I20260929 17:53:36.492319 8455 raft_consensus.cc:2781] T e71e7b3f334b4447a6ea39b42ca38a5b P c8be608a8c8b43e6a7063022a8a7da36 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 17:53:36.492529 8455 raft_consensus.cc:686] T e71e7b3f334b4447a6ea39b42ca38a5b P c8be608a8c8b43e6a7063022a8a7da36 [term 1 LEADER]: Becoming Leader. State: Replica: c8be608a8c8b43e6a7063022a8a7da36, State: Running, Role: LEADER
I20260929 17:53:36.492883 8455 consensus_queue.cc:237] T e71e7b3f334b4447a6ea39b42ca38a5b P c8be608a8c8b43e6a7063022a8a7da36 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8be608a8c8b43e6a7063022a8a7da36" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32913 } } peers { permanent_uuid: "4e821d34d1844842be7365e97204614e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43829 } } peers { permanent_uuid: "ece5590526f94058b07afe5bb610ccfa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40515 } }
I20260929 17:53:36.493224 8095 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e71e7b3f334b4447a6ea39b42ca38a5b" candidate_uuid: "c8be608a8c8b43e6a7063022a8a7da36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ece5590526f94058b07afe5bb610ccfa"
I20260929 17:53:36.493362 8095 raft_consensus.cc:3037] T e71e7b3f334b4447a6ea39b42ca38a5b P ece5590526f94058b07afe5bb610ccfa [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:53:36.494627 8095 raft_consensus.cc:2454] T e71e7b3f334b4447a6ea39b42ca38a5b P ece5590526f94058b07afe5bb610ccfa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8be608a8c8b43e6a7063022a8a7da36 in term 1.
I20260929 17:53:36.496855 7937 catalog_manager.cc:5495] T e71e7b3f334b4447a6ea39b42ca38a5b P c8be608a8c8b43e6a7063022a8a7da36 reported cstate change: term changed from 0 to 1, leader changed from <none> to c8be608a8c8b43e6a7063022a8a7da36 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "c8be608a8c8b43e6a7063022a8a7da36" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8be608a8c8b43e6a7063022a8a7da36" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32913 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4e821d34d1844842be7365e97204614e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43829 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ece5590526f94058b07afe5bb610ccfa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40515 } health_report { overall_health: UNKNOWN } } }
I20260929 17:53:36.523126 8444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:36.519819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43866 (local address 127.0.98.3:32913)
0929 17:53:36.519925 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:36.519928 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:36.519940 (+ 12us) server_negotiation.cc:407] Connection header received
0929 17:53:36.520050 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:36.520052 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:36.520108 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:36.520193 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 17:53:36.520445 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:36.520945 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:36.521799 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:36.522017 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:36.522090 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:36.522355 (+ 265us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 17:53:36.522792 (+ 437us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 17:53:36.522872 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0929 17:53:36.522972 (+ 100us) server_negotiation.cc:299] Negotiation successful
0929 17:53:36.523021 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260929 17:53:36.529258 8095 raft_consensus.cc:1261] T e71e7b3f334b4447a6ea39b42ca38a5b P ece5590526f94058b07afe5bb610ccfa [term 1 FOLLOWER]: Refusing update from remote peer c8be608a8c8b43e6a7063022a8a7da36: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 17:53:36.530112 8235 raft_consensus.cc:1261] T e71e7b3f334b4447a6ea39b42ca38a5b P 4e821d34d1844842be7365e97204614e [term 1 FOLLOWER]: Refusing update from remote peer c8be608a8c8b43e6a7063022a8a7da36: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 17:53:36.530020 8455 consensus_queue.cc:1035] T e71e7b3f334b4447a6ea39b42ca38a5b P c8be608a8c8b43e6a7063022a8a7da36 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ece5590526f94058b07afe5bb610ccfa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 17:53:36.530702 8455 consensus_queue.cc:1035] T e71e7b3f334b4447a6ea39b42ca38a5b P c8be608a8c8b43e6a7063022a8a7da36 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e821d34d1844842be7365e97204614e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 17:53:36.539489 8463 mvcc.cc:205] Tried to move back new op lower bound from 7334725290096398336 to 7334725289958133760. Current Snapshot: MvccSnapshot[applied={T|T < 7334725290096398336}]
I20260929 17:53:36.547026 7937 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51932:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 17:53:36.547220 7937 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51932:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 17:53:36.549582 7937 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1fb78915f2ed49e5b241da7350b145ae: Sending DeleteTablet for 3 replicas of tablet e71e7b3f334b4447a6ea39b42ca38a5b
I20260929 17:53:36.550439 8215 tablet_service.cc:1514] Processing DeleteTablet for tablet e71e7b3f334b4447a6ea39b42ca38a5b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 17:53:36 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45988
I20260929 17:53:36.550474 8075 tablet_service.cc:1514] Processing DeleteTablet for tablet e71e7b3f334b4447a6ea39b42ca38a5b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 17:53:36 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:32840
I20260929 17:53:36.550606 8355 tablet_service.cc:1514] Processing DeleteTablet for tablet e71e7b3f334b4447a6ea39b42ca38a5b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 17:53:36 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43864
I20260929 17:53:36.550979 8473 tablet_replica.cc:330] T e71e7b3f334b4447a6ea39b42ca38a5b P 4e821d34d1844842be7365e97204614e: stopping tablet replica
I20260929 17:53:36.550982 8474 tablet_replica.cc:330] T e71e7b3f334b4447a6ea39b42ca38a5b P c8be608a8c8b43e6a7063022a8a7da36: stopping tablet replica
I20260929 17:53:36.551285 8474 raft_consensus.cc:2229] T e71e7b3f334b4447a6ea39b42ca38a5b P c8be608a8c8b43e6a7063022a8a7da36 [term 1 LEADER]: Raft consensus shutting down.
I20260929 17:53:36.551285 8473 raft_consensus.cc:2229] T e71e7b3f334b4447a6ea39b42ca38a5b P 4e821d34d1844842be7365e97204614e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 17:53:36.551614 8473 raft_consensus.cc:2258] T e71e7b3f334b4447a6ea39b42ca38a5b P 4e821d34d1844842be7365e97204614e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 17:53:36.551627 8474 raft_consensus.cc:2258] T e71e7b3f334b4447a6ea39b42ca38a5b P c8be608a8c8b43e6a7063022a8a7da36 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 17:53:36.551822 7937 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51932:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 17:53:36.552224 7937 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 17:53:36.553182 8474 ts_tablet_manager.cc:1905] T e71e7b3f334b4447a6ea39b42ca38a5b P c8be608a8c8b43e6a7063022a8a7da36: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 17:53:36.553406 8472 tablet_replica.cc:330] T e71e7b3f334b4447a6ea39b42ca38a5b P ece5590526f94058b07afe5bb610ccfa: stopping tablet replica
I20260929 17:53:36.553766 8472 raft_consensus.cc:2229] T e71e7b3f334b4447a6ea39b42ca38a5b P ece5590526f94058b07afe5bb610ccfa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 17:53:36.554126 8472 raft_consensus.cc:2258] T e71e7b3f334b4447a6ea39b42ca38a5b P ece5590526f94058b07afe5bb610ccfa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 17:53:36.554834 8473 ts_tablet_manager.cc:1905] T e71e7b3f334b4447a6ea39b42ca38a5b P 4e821d34d1844842be7365e97204614e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 17:53:36.556114 8474 ts_tablet_manager.cc:1918] T e71e7b3f334b4447a6ea39b42ca38a5b P c8be608a8c8b43e6a7063022a8a7da36: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 17:53:36.556200 8474 log.cc:1196] T e71e7b3f334b4447a6ea39b42ca38a5b P c8be608a8c8b43e6a7063022a8a7da36: Deleting WAL directory at /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704375975685-392-0/minicluster-data/ts-2/wal/wals/e71e7b3f334b4447a6ea39b42ca38a5b
I20260929 17:53:36.556448 8474 ts_tablet_manager.cc:1939] T e71e7b3f334b4447a6ea39b42ca38a5b P c8be608a8c8b43e6a7063022a8a7da36: Deleting consensus metadata
I20260929 17:53:36.557206 8075 tablet_service.cc:1467] Processing CreateTablet for tablet dd520d47442846f183e4292f0df97d64 (DEFAULT_TABLE table=test-table [id=420b3c3ab44a4bf7aa830edb0fc4c7c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 17:53:36.557466 8075 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd520d47442846f183e4292f0df97d64. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:53:36.557545 7925 catalog_manager.cc:4841] TS c8be608a8c8b43e6a7063022a8a7da36 (127.0.98.3:32913): tablet e71e7b3f334b4447a6ea39b42ca38a5b (table test-table [id=4f2c328f95bd4e87bc0c6ef87d8bb1f2]) successfully deleted
I20260929 17:53:36.557567 8215 tablet_service.cc:1467] Processing CreateTablet for tablet dd520d47442846f183e4292f0df97d64 (DEFAULT_TABLE table=test-table [id=420b3c3ab44a4bf7aa830edb0fc4c7c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 17:53:36.557808 8215 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd520d47442846f183e4292f0df97d64. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:53:36.558816 8472 ts_tablet_manager.cc:1905] T e71e7b3f334b4447a6ea39b42ca38a5b P ece5590526f94058b07afe5bb610ccfa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 17:53:36.559865 8451 tablet_bootstrap.cc:492] T dd520d47442846f183e4292f0df97d64 P ece5590526f94058b07afe5bb610ccfa: Bootstrap starting.
I20260929 17:53:36.560034 8449 tablet_bootstrap.cc:492] T dd520d47442846f183e4292f0df97d64 P 4e821d34d1844842be7365e97204614e: Bootstrap starting.
I20260929 17:53:36.560258 8355 tablet_service.cc:1467] Processing CreateTablet for tablet dd520d47442846f183e4292f0df97d64 (DEFAULT_TABLE table=test-table [id=420b3c3ab44a4bf7aa830edb0fc4c7c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 17:53:36.560491 8355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd520d47442846f183e4292f0df97d64. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:53:36.560768 8451 tablet_bootstrap.cc:654] T dd520d47442846f183e4292f0df97d64 P ece5590526f94058b07afe5bb610ccfa: Neither blocks nor log segments found. Creating new log.
I20260929 17:53:36.560932 8449 tablet_bootstrap.cc:654] T dd520d47442846f183e4292f0df97d64 P 4e821d34d1844842be7365e97204614e: Neither blocks nor log segments found. Creating new log.
I20260929 17:53:36.560979 8472 ts_tablet_manager.cc:1918] T e71e7b3f334b4447a6ea39b42ca38a5b P ece5590526f94058b07afe5bb610ccfa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 17:53:36.561081 8472 log.cc:1196] T e71e7b3f334b4447a6ea39b42ca38a5b P ece5590526f94058b07afe5bb610ccfa: Deleting WAL directory at /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704375975685-392-0/minicluster-data/ts-0/wal/wals/e71e7b3f334b4447a6ea39b42ca38a5b
I20260929 17:53:36.561750 8472 ts_tablet_manager.cc:1939] T e71e7b3f334b4447a6ea39b42ca38a5b P ece5590526f94058b07afe5bb610ccfa: Deleting consensus metadata
I20260929 17:53:36.562898 8450 tablet_bootstrap.cc:492] T dd520d47442846f183e4292f0df97d64 P c8be608a8c8b43e6a7063022a8a7da36: Bootstrap starting.
I20260929 17:53:36.562907 7922 catalog_manager.cc:4841] TS ece5590526f94058b07afe5bb610ccfa (127.0.98.1:40515): tablet e71e7b3f334b4447a6ea39b42ca38a5b (table test-table [id=4f2c328f95bd4e87bc0c6ef87d8bb1f2]) successfully deleted
I20260929 17:53:36.563588 8451 tablet_bootstrap.cc:492] T dd520d47442846f183e4292f0df97d64 P ece5590526f94058b07afe5bb610ccfa: No bootstrap required, opened a new log
I20260929 17:53:36.563669 8451 ts_tablet_manager.cc:1397] T dd520d47442846f183e4292f0df97d64 P ece5590526f94058b07afe5bb610ccfa: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260929 17:53:36.563795 8450 tablet_bootstrap.cc:654] T dd520d47442846f183e4292f0df97d64 P c8be608a8c8b43e6a7063022a8a7da36: Neither blocks nor log segments found. Creating new log.
I20260929 17:53:36.564064 8451 raft_consensus.cc:348] T dd520d47442846f183e4292f0df97d64 P ece5590526f94058b07afe5bb610ccfa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ece5590526f94058b07afe5bb610ccfa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40515 } } peers { permanent_uuid: "4e821d34d1844842be7365e97204614e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43829 } } peers { permanent_uuid: "c8be608a8c8b43e6a7063022a8a7da36" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32913 } }
I20260929 17:53:36.564157 8451 raft_consensus.cc:374] T dd520d47442846f183e4292f0df97d64 P ece5590526f94058b07afe5bb610ccfa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:53:36.564188 8451 raft_consensus.cc:729] T dd520d47442846f183e4292f0df97d64 P ece5590526f94058b07afe5bb610ccfa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ece5590526f94058b07afe5bb610ccfa, State: Initialized, Role: FOLLOWER
I20260929 17:53:36.564303 8451 consensus_queue.cc:260] T dd520d47442846f183e4292f0df97d64 P ece5590526f94058b07afe5bb610ccfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ece5590526f94058b07afe5bb610ccfa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40515 } } peers { permanent_uuid: "4e821d34d1844842be7365e97204614e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43829 } } peers { permanent_uuid: "c8be608a8c8b43e6a7063022a8a7da36" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32913 } }
I20260929 17:53:36.564543 8451 ts_tablet_manager.cc:1428] T dd520d47442846f183e4292f0df97d64 P ece5590526f94058b07afe5bb610ccfa: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 17:53:36.564903 8473 ts_tablet_manager.cc:1918] T e71e7b3f334b4447a6ea39b42ca38a5b P 4e821d34d1844842be7365e97204614e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 17:53:36.564985 8473 log.cc:1196] T e71e7b3f334b4447a6ea39b42ca38a5b P 4e821d34d1844842be7365e97204614e: Deleting WAL directory at /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704375975685-392-0/minicluster-data/ts-1/wal/wals/e71e7b3f334b4447a6ea39b42ca38a5b
I20260929 17:53:36.565285 8473 ts_tablet_manager.cc:1939] T e71e7b3f334b4447a6ea39b42ca38a5b P 4e821d34d1844842be7365e97204614e: Deleting consensus metadata
I20260929 17:53:36.566399 7925 catalog_manager.cc:4841] TS 4e821d34d1844842be7365e97204614e (127.0.98.2:43829): tablet e71e7b3f334b4447a6ea39b42ca38a5b (table test-table [id=4f2c328f95bd4e87bc0c6ef87d8bb1f2]) successfully deleted
I20260929 17:53:36.566953 8449 tablet_bootstrap.cc:492] T dd520d47442846f183e4292f0df97d64 P 4e821d34d1844842be7365e97204614e: No bootstrap required, opened a new log
I20260929 17:53:36.567030 8449 ts_tablet_manager.cc:1397] T dd520d47442846f183e4292f0df97d64 P 4e821d34d1844842be7365e97204614e: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.001s
I20260929 17:53:36.567212 8450 tablet_bootstrap.cc:492] T dd520d47442846f183e4292f0df97d64 P c8be608a8c8b43e6a7063022a8a7da36: No bootstrap required, opened a new log
I20260929 17:53:36.567286 8450 ts_tablet_manager.cc:1397] T dd520d47442846f183e4292f0df97d64 P c8be608a8c8b43e6a7063022a8a7da36: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20260929 17:53:36.567489 8449 raft_consensus.cc:348] T dd520d47442846f183e4292f0df97d64 P 4e821d34d1844842be7365e97204614e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ece5590526f94058b07afe5bb610ccfa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40515 } } peers { permanent_uuid: "4e821d34d1844842be7365e97204614e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43829 } } peers { permanent_uuid: "c8be608a8c8b43e6a7063022a8a7da36" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32913 } }
I20260929 17:53:36.567595 8449 raft_consensus.cc:374] T dd520d47442846f183e4292f0df97d64 P 4e821d34d1844842be7365e97204614e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:53:36.567626 8449 raft_consensus.cc:729] T dd520d47442846f183e4292f0df97d64 P 4e821d34d1844842be7365e97204614e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e821d34d1844842be7365e97204614e, State: Initialized, Role: FOLLOWER
I20260929 17:53:36.567711 8450 raft_consensus.cc:348] T dd520d47442846f183e4292f0df97d64 P c8be608a8c8b43e6a7063022a8a7da36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ece5590526f94058b07afe5bb610ccfa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40515 } } peers { permanent_uuid: "4e821d34d1844842be7365e97204614e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43829 } } peers { permanent_uuid: "c8be608a8c8b43e6a7063022a8a7da36" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32913 } }
I20260929 17:53:36.567816 8450 raft_consensus.cc:374] T dd520d47442846f183e4292f0df97d64 P c8be608a8c8b43e6a7063022a8a7da36 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:53:36.567768 8449 consensus_queue.cc:260] T dd520d47442846f183e4292f0df97d64 P 4e821d34d1844842be7365e97204614e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ece5590526f94058b07afe5bb610ccfa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40515 } } peers { permanent_uuid: "4e821d34d1844842be7365e97204614e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43829 } } peers { permanent_uuid: "c8be608a8c8b43e6a7063022a8a7da36" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32913 } }
I20260929 17:53:36.567845 8450 raft_consensus.cc:729] T dd520d47442846f183e4292f0df97d64 P c8be608a8c8b43e6a7063022a8a7da36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8be608a8c8b43e6a7063022a8a7da36, State: Initialized, Role: FOLLOWER
I20260929 17:53:36.567952 8450 consensus_queue.cc:260] T dd520d47442846f183e4292f0df97d64 P c8be608a8c8b43e6a7063022a8a7da36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ece5590526f94058b07afe5bb610ccfa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40515 } } peers { permanent_uuid: "4e821d34d1844842be7365e97204614e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43829 } } peers { permanent_uuid: "c8be608a8c8b43e6a7063022a8a7da36" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32913 } }
I20260929 17:53:36.568013 8449 ts_tablet_manager.cc:1428] T dd520d47442846f183e4292f0df97d64 P 4e821d34d1844842be7365e97204614e: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260929 17:53:36.568183 8450 ts_tablet_manager.cc:1428] T dd520d47442846f183e4292f0df97d64 P c8be608a8c8b43e6a7063022a8a7da36: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
W20260929 17:53:36.613364 8422 tablet.cc:2369] T dd520d47442846f183e4292f0df97d64 P c8be608a8c8b43e6a7063022a8a7da36: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 17:53:36.650728 8282 tablet.cc:2369] T dd520d47442846f183e4292f0df97d64 P 4e821d34d1844842be7365e97204614e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 17:53:36.683692 8142 tablet.cc:2369] T dd520d47442846f183e4292f0df97d64 P ece5590526f94058b07afe5bb610ccfa: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 17:53:36.685866 8469 raft_consensus.cc:482] T dd520d47442846f183e4292f0df97d64 P c8be608a8c8b43e6a7063022a8a7da36 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 17:53:36.686002 8469 raft_consensus.cc:504] T dd520d47442846f183e4292f0df97d64 P c8be608a8c8b43e6a7063022a8a7da36 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ece5590526f94058b07afe5bb610ccfa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40515 } } peers { permanent_uuid: "4e821d34d1844842be7365e97204614e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43829 } } peers { permanent_uuid: "c8be608a8c8b43e6a7063022a8a7da36" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32913 } }
I20260929 17:53:36.686441 8469 leader_election.cc:288] T dd520d47442846f183e4292f0df97d64 P c8be608a8c8b43e6a7063022a8a7da36 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ece5590526f94058b07afe5bb610ccfa (127.0.98.1:40515), 4e821d34d1844842be7365e97204614e (127.0.98.2:43829)
I20260929 17:53:36.686846 8095 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd520d47442846f183e4292f0df97d64" candidate_uuid: "c8be608a8c8b43e6a7063022a8a7da36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ece5590526f94058b07afe5bb610ccfa" is_pre_election: true
I20260929 17:53:36.686903 8235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd520d47442846f183e4292f0df97d64" candidate_uuid: "c8be608a8c8b43e6a7063022a8a7da36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e821d34d1844842be7365e97204614e" is_pre_election: true
I20260929 17:53:36.686990 8095 raft_consensus.cc:2454] T dd520d47442846f183e4292f0df97d64 P ece5590526f94058b07afe5bb610ccfa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c8be608a8c8b43e6a7063022a8a7da36 in term 0.
I20260929 17:53:36.687074 8235 raft_consensus.cc:2454] T dd520d47442846f183e4292f0df97d64 P 4e821d34d1844842be7365e97204614e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c8be608a8c8b43e6a7063022a8a7da36 in term 0.
I20260929 17:53:36.687294 8306 leader_election.cc:302] T dd520d47442846f183e4292f0df97d64 P c8be608a8c8b43e6a7063022a8a7da36 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c8be608a8c8b43e6a7063022a8a7da36, ece5590526f94058b07afe5bb610ccfa; no voters:
I20260929 17:53:36.687453 8469 raft_consensus.cc:2781] T dd520d47442846f183e4292f0df97d64 P c8be608a8c8b43e6a7063022a8a7da36 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 17:53:36.687510 8469 raft_consensus.cc:482] T dd520d47442846f183e4292f0df97d64 P c8be608a8c8b43e6a7063022a8a7da36 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 17:53:36.687546 8469 raft_consensus.cc:3037] T dd520d47442846f183e4292f0df97d64 P c8be608a8c8b43e6a7063022a8a7da36 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:53:36.688524 8469 raft_consensus.cc:504] T dd520d47442846f183e4292f0df97d64 P c8be608a8c8b43e6a7063022a8a7da36 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ece5590526f94058b07afe5bb610ccfa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40515 } } peers { permanent_uuid: "4e821d34d1844842be7365e97204614e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43829 } } peers { permanent_uuid: "c8be608a8c8b43e6a7063022a8a7da36" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32913 } }
I20260929 17:53:36.688863 8469 leader_election.cc:288] T dd520d47442846f183e4292f0df97d64 P c8be608a8c8b43e6a7063022a8a7da36 [CANDIDATE]: Term 1 election: Requested vote from peers ece5590526f94058b07afe5bb610ccfa (127.0.98.1:40515), 4e821d34d1844842be7365e97204614e (127.0.98.2:43829)
I20260929 17:53:36.689225 8235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd520d47442846f183e4292f0df97d64" candidate_uuid: "c8be608a8c8b43e6a7063022a8a7da36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e821d34d1844842be7365e97204614e"
I20260929 17:53:36.689241 8095 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd520d47442846f183e4292f0df97d64" candidate_uuid: "c8be608a8c8b43e6a7063022a8a7da36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ece5590526f94058b07afe5bb610ccfa"
I20260929 17:53:36.689333 8235 raft_consensus.cc:3037] T dd520d47442846f183e4292f0df97d64 P 4e821d34d1844842be7365e97204614e [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:53:36.689369 8095 raft_consensus.cc:3037] T dd520d47442846f183e4292f0df97d64 P ece5590526f94058b07afe5bb610ccfa [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:53:36.690264 8235 raft_consensus.cc:2454] T dd520d47442846f183e4292f0df97d64 P 4e821d34d1844842be7365e97204614e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8be608a8c8b43e6a7063022a8a7da36 in term 1.
I20260929 17:53:36.690528 8095 raft_consensus.cc:2454] T dd520d47442846f183e4292f0df97d64 P ece5590526f94058b07afe5bb610ccfa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8be608a8c8b43e6a7063022a8a7da36 in term 1.
I20260929 17:53:36.690580 8309 leader_election.cc:302] T dd520d47442846f183e4292f0df97d64 P c8be608a8c8b43e6a7063022a8a7da36 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e821d34d1844842be7365e97204614e, c8be608a8c8b43e6a7063022a8a7da36; no voters:
I20260929 17:53:36.690831 8469 raft_consensus.cc:2781] T dd520d47442846f183e4292f0df97d64 P c8be608a8c8b43e6a7063022a8a7da36 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 17:53:36.690933 8469 raft_consensus.cc:686] T dd520d47442846f183e4292f0df97d64 P c8be608a8c8b43e6a7063022a8a7da36 [term 1 LEADER]: Becoming Leader. State: Replica: c8be608a8c8b43e6a7063022a8a7da36, State: Running, Role: LEADER
I20260929 17:53:36.691076 8469 consensus_queue.cc:237] T dd520d47442846f183e4292f0df97d64 P c8be608a8c8b43e6a7063022a8a7da36 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ece5590526f94058b07afe5bb610ccfa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40515 } } peers { permanent_uuid: "4e821d34d1844842be7365e97204614e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43829 } } peers { permanent_uuid: "c8be608a8c8b43e6a7063022a8a7da36" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32913 } }
I20260929 17:53:36.692706 7936 catalog_manager.cc:5495] T dd520d47442846f183e4292f0df97d64 P c8be608a8c8b43e6a7063022a8a7da36 reported cstate change: term changed from 0 to 1, leader changed from <none> to c8be608a8c8b43e6a7063022a8a7da36 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "c8be608a8c8b43e6a7063022a8a7da36" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ece5590526f94058b07afe5bb610ccfa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e821d34d1844842be7365e97204614e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43829 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8be608a8c8b43e6a7063022a8a7da36" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32913 } health_report { overall_health: HEALTHY } } }
Sep 29 17:53:36 dist-test-slave-53nh krb5kdc[7893](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704415, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260929 17:53:36.742311 8484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:36.734649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44965 (local address 127.0.0.1:51938)
0929 17:53:36.734942 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:36.734954 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 17:53:36.735062 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:53:36.735282 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:53:36.735285 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:53:36.735693 (+ 408us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:53:36.735782 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:36.735789 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:36.736488 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:36.736491 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:36.737107 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:36.737116 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:36.737513 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:36.737516 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:36.737585 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:36.738115 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:53:36.738127 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:53:36.739425 (+ 1298us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:53:36.740776 (+ 1351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:53:36.740785 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:53:36.740792 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:53:36.741096 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:53:36.741371 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:53:36.741374 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:53:36.741375 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:53:36.741490 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:53:36.741816 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:53:36.741822 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:53:36.741931 (+ 109us) client_negotiation.cc:769] Sending connection context
0929 17:53:36.742027 (+ 96us) client_negotiation.cc:240] Negotiation successful
0929 17:53:36.742153 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":131,"threads_started":1}
I20260929 17:53:36.742329 7987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:36.734708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51938 (local address 127.0.98.62:44965)
0929 17:53:36.734842 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:36.734847 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:36.735020 (+ 173us) server_negotiation.cc:407] Connection header received
0929 17:53:36.735151 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:36.735154 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:36.735195 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:36.735253 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:53:36.735885 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:36.736397 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:36.737208 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:36.737409 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:36.737511 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:36.739544 (+ 2033us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:53:36.739548 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:53:36.739550 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:53:36.739570 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:53:36.740686 (+ 1116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:53:36.741179 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:53:36.741182 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:53:36.741184 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:53:36.741251 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:53:36.741566 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:53:36.741569 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:53:36.741570 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:53:36.741730 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:53:36.741799 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0929 17:53:36.742061 (+ 262us) server_negotiation.cc:299] Negotiation successful
0929 17:53:36.742210 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":49,"server-negotiator.queue_time_us":54}
I20260929 17:53:36.746158 7936 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51938:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260929 17:53:36.746795 7936 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 17:53:36.751324 8215 tablet_service.cc:1467] Processing CreateTablet for tablet bbe5590ef0094985a27a4ba2caf10534 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=62c579702df94a3aadeed2151aae1968]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 17:53:36.751302 8075 tablet_service.cc:1467] Processing CreateTablet for tablet bbe5590ef0094985a27a4ba2caf10534 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=62c579702df94a3aadeed2151aae1968]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 17:53:36.751705 8075 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bbe5590ef0094985a27a4ba2caf10534. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:53:36.751704 8215 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bbe5590ef0094985a27a4ba2caf10534. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:53:36.753561 8355 tablet_service.cc:1467] Processing CreateTablet for tablet bbe5590ef0094985a27a4ba2caf10534 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=62c579702df94a3aadeed2151aae1968]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 17:53:36.754009 8355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bbe5590ef0094985a27a4ba2caf10534. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:53:36.754696 8449 tablet_bootstrap.cc:492] T bbe5590ef0094985a27a4ba2caf10534 P 4e821d34d1844842be7365e97204614e: Bootstrap starting.
I20260929 17:53:36.755533 8449 tablet_bootstrap.cc:654] T bbe5590ef0094985a27a4ba2caf10534 P 4e821d34d1844842be7365e97204614e: Neither blocks nor log segments found. Creating new log.
I20260929 17:53:36.756050 8451 tablet_bootstrap.cc:492] T bbe5590ef0094985a27a4ba2caf10534 P ece5590526f94058b07afe5bb610ccfa: Bootstrap starting.
I20260929 17:53:36.756397 8450 tablet_bootstrap.cc:492] T bbe5590ef0094985a27a4ba2caf10534 P c8be608a8c8b43e6a7063022a8a7da36: Bootstrap starting.
I20260929 17:53:36.756899 8451 tablet_bootstrap.cc:654] T bbe5590ef0094985a27a4ba2caf10534 P ece5590526f94058b07afe5bb610ccfa: Neither blocks nor log segments found. Creating new log.
I20260929 17:53:36.757153 8450 tablet_bootstrap.cc:654] T bbe5590ef0094985a27a4ba2caf10534 P c8be608a8c8b43e6a7063022a8a7da36: Neither blocks nor log segments found. Creating new log.
I20260929 17:53:36.758119 8451 tablet_bootstrap.cc:492] T bbe5590ef0094985a27a4ba2caf10534 P ece5590526f94058b07afe5bb610ccfa: No bootstrap required, opened a new log
I20260929 17:53:36.758204 8451 ts_tablet_manager.cc:1397] T bbe5590ef0094985a27a4ba2caf10534 P ece5590526f94058b07afe5bb610ccfa: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260929 17:53:36.758388 8449 tablet_bootstrap.cc:492] T bbe5590ef0094985a27a4ba2caf10534 P 4e821d34d1844842be7365e97204614e: No bootstrap required, opened a new log
I20260929 17:53:36.758479 8449 ts_tablet_manager.cc:1397] T bbe5590ef0094985a27a4ba2caf10534 P 4e821d34d1844842be7365e97204614e: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20260929 17:53:36.758615 8450 tablet_bootstrap.cc:492] T bbe5590ef0094985a27a4ba2caf10534 P c8be608a8c8b43e6a7063022a8a7da36: No bootstrap required, opened a new log
I20260929 17:53:36.758695 8450 ts_tablet_manager.cc:1397] T bbe5590ef0094985a27a4ba2caf10534 P c8be608a8c8b43e6a7063022a8a7da36: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 17:53:36.758625 8451 raft_consensus.cc:348] T bbe5590ef0094985a27a4ba2caf10534 P ece5590526f94058b07afe5bb610ccfa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ece5590526f94058b07afe5bb610ccfa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40515 } } peers { permanent_uuid: "4e821d34d1844842be7365e97204614e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43829 } } peers { permanent_uuid: "c8be608a8c8b43e6a7063022a8a7da36" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32913 } }
I20260929 17:53:36.758739 8451 raft_consensus.cc:374] T bbe5590ef0094985a27a4ba2caf10534 P ece5590526f94058b07afe5bb610ccfa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:53:36.758774 8451 raft_consensus.cc:729] T bbe5590ef0094985a27a4ba2caf10534 P ece5590526f94058b07afe5bb610ccfa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ece5590526f94058b07afe5bb610ccfa, State: Initialized, Role: FOLLOWER
I20260929 17:53:36.758908 8451 consensus_queue.cc:260] T bbe5590ef0094985a27a4ba2caf10534 P ece5590526f94058b07afe5bb610ccfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ece5590526f94058b07afe5bb610ccfa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40515 } } peers { permanent_uuid: "4e821d34d1844842be7365e97204614e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43829 } } peers { permanent_uuid: "c8be608a8c8b43e6a7063022a8a7da36" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32913 } }
I20260929 17:53:36.758934 8449 raft_consensus.cc:348] T bbe5590ef0094985a27a4ba2caf10534 P 4e821d34d1844842be7365e97204614e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ece5590526f94058b07afe5bb610ccfa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40515 } } peers { permanent_uuid: "4e821d34d1844842be7365e97204614e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43829 } } peers { permanent_uuid: "c8be608a8c8b43e6a7063022a8a7da36" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32913 } }
I20260929 17:53:36.759047 8449 raft_consensus.cc:374] T bbe5590ef0094985a27a4ba2caf10534 P 4e821d34d1844842be7365e97204614e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:53:36.759089 8449 raft_consensus.cc:729] T bbe5590ef0094985a27a4ba2caf10534 P 4e821d34d1844842be7365e97204614e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e821d34d1844842be7365e97204614e, State: Initialized, Role: FOLLOWER
I20260929 17:53:36.759054 8450 raft_consensus.cc:348] T bbe5590ef0094985a27a4ba2caf10534 P c8be608a8c8b43e6a7063022a8a7da36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ece5590526f94058b07afe5bb610ccfa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40515 } } peers { permanent_uuid: "4e821d34d1844842be7365e97204614e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43829 } } peers { permanent_uuid: "c8be608a8c8b43e6a7063022a8a7da36" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32913 } }
I20260929 17:53:36.759155 8450 raft_consensus.cc:374] T bbe5590ef0094985a27a4ba2caf10534 P c8be608a8c8b43e6a7063022a8a7da36 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:53:36.759178 8451 ts_tablet_manager.cc:1428] T bbe5590ef0094985a27a4ba2caf10534 P ece5590526f94058b07afe5bb610ccfa: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260929 17:53:36.759200 8450 raft_consensus.cc:729] T bbe5590ef0094985a27a4ba2caf10534 P c8be608a8c8b43e6a7063022a8a7da36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8be608a8c8b43e6a7063022a8a7da36, State: Initialized, Role: FOLLOWER
I20260929 17:53:36.759207 8449 consensus_queue.cc:260] T bbe5590ef0094985a27a4ba2caf10534 P 4e821d34d1844842be7365e97204614e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ece5590526f94058b07afe5bb610ccfa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40515 } } peers { permanent_uuid: "4e821d34d1844842be7365e97204614e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43829 } } peers { permanent_uuid: "c8be608a8c8b43e6a7063022a8a7da36" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32913 } }
I20260929 17:53:36.759315 8450 consensus_queue.cc:260] T bbe5590ef0094985a27a4ba2caf10534 P c8be608a8c8b43e6a7063022a8a7da36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ece5590526f94058b07afe5bb610ccfa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40515 } } peers { permanent_uuid: "4e821d34d1844842be7365e97204614e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43829 } } peers { permanent_uuid: "c8be608a8c8b43e6a7063022a8a7da36" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32913 } }
I20260929 17:53:36.759449 8449 ts_tablet_manager.cc:1428] T bbe5590ef0094985a27a4ba2caf10534 P 4e821d34d1844842be7365e97204614e: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20260929 17:53:36.759546 8450 ts_tablet_manager.cc:1428] T bbe5590ef0094985a27a4ba2caf10534 P c8be608a8c8b43e6a7063022a8a7da36: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 17:53:36.759704 8456 tablet_replica.cc:439] T bbe5590ef0094985a27a4ba2caf10534 P 4e821d34d1844842be7365e97204614e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ece5590526f94058b07afe5bb610ccfa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40515 } } peers { permanent_uuid: "4e821d34d1844842be7365e97204614e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43829 } } peers { permanent_uuid: "c8be608a8c8b43e6a7063022a8a7da36" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32913 } } }
I20260929 17:53:36.759747 8457 tablet_replica.cc:439] T bbe5590ef0094985a27a4ba2caf10534 P ece5590526f94058b07afe5bb610ccfa: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ece5590526f94058b07afe5bb610ccfa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40515 } } peers { permanent_uuid: "4e821d34d1844842be7365e97204614e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43829 } } peers { permanent_uuid: "c8be608a8c8b43e6a7063022a8a7da36" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32913 } } }
I20260929 17:53:36.759868 8456 tablet_replica.cc:442] T bbe5590ef0094985a27a4ba2caf10534 P 4e821d34d1844842be7365e97204614e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 17:53:36.759867 8457 tablet_replica.cc:442] T bbe5590ef0094985a27a4ba2caf10534 P ece5590526f94058b07afe5bb610ccfa: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 17:53:36.760326 8469 tablet_replica.cc:439] T bbe5590ef0094985a27a4ba2caf10534 P c8be608a8c8b43e6a7063022a8a7da36: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ece5590526f94058b07afe5bb610ccfa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40515 } } peers { permanent_uuid: "4e821d34d1844842be7365e97204614e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43829 } } peers { permanent_uuid: "c8be608a8c8b43e6a7063022a8a7da36" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32913 } } }
I20260929 17:53:36.760439 8469 tablet_replica.cc:442] T bbe5590ef0094985a27a4ba2caf10534 P c8be608a8c8b43e6a7063022a8a7da36: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 17:53:36.851137 8456 raft_consensus.cc:482] T bbe5590ef0094985a27a4ba2caf10534 P 4e821d34d1844842be7365e97204614e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 17:53:36.851271 8456 raft_consensus.cc:504] T bbe5590ef0094985a27a4ba2caf10534 P 4e821d34d1844842be7365e97204614e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ece5590526f94058b07afe5bb610ccfa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40515 } } peers { permanent_uuid: "4e821d34d1844842be7365e97204614e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43829 } } peers { permanent_uuid: "c8be608a8c8b43e6a7063022a8a7da36" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32913 } }
I20260929 17:53:36.852358 8456 leader_election.cc:288] T bbe5590ef0094985a27a4ba2caf10534 P 4e821d34d1844842be7365e97204614e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ece5590526f94058b07afe5bb610ccfa (127.0.98.1:40515), c8be608a8c8b43e6a7063022a8a7da36 (127.0.98.3:32913)
I20260929 17:53:36.857226 8444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:36.852812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59639 (local address 127.0.98.3:32913)
0929 17:53:36.852910 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:36.852913 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:36.853016 (+ 103us) server_negotiation.cc:407] Connection header received
0929 17:53:36.853118 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:36.853121 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:36.853162 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:36.853228 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:53:36.853540 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:36.854317 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:36.855971 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:36.856595 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:36.856708 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:36.856768 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0929 17:53:36.857027 (+ 259us) server_negotiation.cc:299] Negotiation successful
0929 17:53:36.857095 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260929 17:53:36.857227 8447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:36.852657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59951 (local address 127.0.98.1:40515)
0929 17:53:36.852795 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:36.852799 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:36.852876 (+ 77us) server_negotiation.cc:407] Connection header received
0929 17:53:36.853084 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:36.853087 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:36.853136 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:36.853223 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:53:36.853540 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:36.854327 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:36.855971 (+ 1644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:36.856595 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:36.856702 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:36.856766 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0929 17:53:36.857027 (+ 261us) server_negotiation.cc:299] Negotiation successful
0929 17:53:36.857101 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 17:53:36.857265 8489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:36.852607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:32913 (local address 127.0.98.2:59639)
0929 17:53:36.852937 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:36.852948 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 17:53:36.853046 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:53:36.853238 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:53:36.853240 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:53:36.853280 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:53:36.853447 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:36.853452 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:36.854418 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:36.854422 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:36.855847 (+ 1425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:36.855858 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:36.856729 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:36.856734 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:36.856874 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:36.856917 (+ 43us) client_negotiation.cc:769] Sending connection context
0929 17:53:36.856985 (+ 68us) client_negotiation.cc:240] Negotiation successful
0929 17:53:36.857087 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"mutex_wait_us":155,"spinlock_wait_cycles":5504,"thread_start_us":81,"threads_started":1}
I20260929 17:53:36.857270 8488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:36.852523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40515 (local address 127.0.98.2:59951)
0929 17:53:36.852792 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:36.852807 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 17:53:36.852952 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:53:36.853239 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:53:36.853241 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:53:36.853278 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:53:36.853441 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:36.853447 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:36.854418 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:36.854422 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:36.855847 (+ 1425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:36.855858 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:36.856729 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:36.856734 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:36.856870 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:36.856910 (+ 40us) client_negotiation.cc:769] Sending connection context
0929 17:53:36.856985 (+ 75us) client_negotiation.cc:240] Negotiation successful
0929 17:53:36.857119 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":204,"mutex_wait_us":122,"spinlock_wait_cycles":30208,"thread_start_us":91,"threads_started":1}
I20260929 17:53:36.857762 8375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbe5590ef0094985a27a4ba2caf10534" candidate_uuid: "4e821d34d1844842be7365e97204614e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8be608a8c8b43e6a7063022a8a7da36" is_pre_election: true
I20260929 17:53:36.857962 8095 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbe5590ef0094985a27a4ba2caf10534" candidate_uuid: "4e821d34d1844842be7365e97204614e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ece5590526f94058b07afe5bb610ccfa" is_pre_election: true
I20260929 17:53:36.857985 8375 raft_consensus.cc:2454] T bbe5590ef0094985a27a4ba2caf10534 P c8be608a8c8b43e6a7063022a8a7da36 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e821d34d1844842be7365e97204614e in term 0.
I20260929 17:53:36.858100 8095 raft_consensus.cc:2454] T bbe5590ef0094985a27a4ba2caf10534 P ece5590526f94058b07afe5bb610ccfa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e821d34d1844842be7365e97204614e in term 0.
I20260929 17:53:36.858410 8166 leader_election.cc:302] T bbe5590ef0094985a27a4ba2caf10534 P 4e821d34d1844842be7365e97204614e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e821d34d1844842be7365e97204614e, ece5590526f94058b07afe5bb610ccfa; no voters:
I20260929 17:53:36.858652 8456 raft_consensus.cc:2781] T bbe5590ef0094985a27a4ba2caf10534 P 4e821d34d1844842be7365e97204614e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 17:53:36.858731 8456 raft_consensus.cc:482] T bbe5590ef0094985a27a4ba2caf10534 P 4e821d34d1844842be7365e97204614e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 17:53:36.858778 8456 raft_consensus.cc:3037] T bbe5590ef0094985a27a4ba2caf10534 P 4e821d34d1844842be7365e97204614e [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:53:36.859692 8456 raft_consensus.cc:504] T bbe5590ef0094985a27a4ba2caf10534 P 4e821d34d1844842be7365e97204614e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ece5590526f94058b07afe5bb610ccfa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40515 } } peers { permanent_uuid: "4e821d34d1844842be7365e97204614e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43829 } } peers { permanent_uuid: "c8be608a8c8b43e6a7063022a8a7da36" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32913 } }
I20260929 17:53:36.860005 8456 leader_election.cc:288] T bbe5590ef0094985a27a4ba2caf10534 P 4e821d34d1844842be7365e97204614e [CANDIDATE]: Term 1 election: Requested vote from peers ece5590526f94058b07afe5bb610ccfa (127.0.98.1:40515), c8be608a8c8b43e6a7063022a8a7da36 (127.0.98.3:32913)
I20260929 17:53:36.860311 8375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbe5590ef0094985a27a4ba2caf10534" candidate_uuid: "4e821d34d1844842be7365e97204614e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8be608a8c8b43e6a7063022a8a7da36"
I20260929 17:53:36.860324 8095 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbe5590ef0094985a27a4ba2caf10534" candidate_uuid: "4e821d34d1844842be7365e97204614e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ece5590526f94058b07afe5bb610ccfa"
I20260929 17:53:36.860421 8095 raft_consensus.cc:3037] T bbe5590ef0094985a27a4ba2caf10534 P ece5590526f94058b07afe5bb610ccfa [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:53:36.860426 8375 raft_consensus.cc:3037] T bbe5590ef0094985a27a4ba2caf10534 P c8be608a8c8b43e6a7063022a8a7da36 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:53:36.861270 8095 raft_consensus.cc:2454] T bbe5590ef0094985a27a4ba2caf10534 P ece5590526f94058b07afe5bb610ccfa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e821d34d1844842be7365e97204614e in term 1.
I20260929 17:53:36.861431 8375 raft_consensus.cc:2454] T bbe5590ef0094985a27a4ba2caf10534 P c8be608a8c8b43e6a7063022a8a7da36 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e821d34d1844842be7365e97204614e in term 1.
I20260929 17:53:36.861502 8166 leader_election.cc:302] T bbe5590ef0094985a27a4ba2caf10534 P 4e821d34d1844842be7365e97204614e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e821d34d1844842be7365e97204614e, ece5590526f94058b07afe5bb610ccfa; no voters:
I20260929 17:53:36.861650 8456 raft_consensus.cc:2781] T bbe5590ef0094985a27a4ba2caf10534 P 4e821d34d1844842be7365e97204614e [term 1 FOLLOWER]: Leader election won for term 1
I20260929 17:53:36.861835 8456 raft_consensus.cc:686] T bbe5590ef0094985a27a4ba2caf10534 P 4e821d34d1844842be7365e97204614e [term 1 LEADER]: Becoming Leader. State: Replica: 4e821d34d1844842be7365e97204614e, State: Running, Role: LEADER
I20260929 17:53:36.862068 8456 consensus_queue.cc:237] T bbe5590ef0094985a27a4ba2caf10534 P 4e821d34d1844842be7365e97204614e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ece5590526f94058b07afe5bb610ccfa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40515 } } peers { permanent_uuid: "4e821d34d1844842be7365e97204614e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43829 } } peers { permanent_uuid: "c8be608a8c8b43e6a7063022a8a7da36" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32913 } }
I20260929 17:53:36.863247 8490 tablet_replica.cc:439] T bbe5590ef0094985a27a4ba2caf10534 P 4e821d34d1844842be7365e97204614e: TxnStatusTablet state changed. Reason: New leader 4e821d34d1844842be7365e97204614e. Latest consensus state: current_term: 1 leader_uuid: "4e821d34d1844842be7365e97204614e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ece5590526f94058b07afe5bb610ccfa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40515 } } peers { permanent_uuid: "4e821d34d1844842be7365e97204614e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43829 } } peers { permanent_uuid: "c8be608a8c8b43e6a7063022a8a7da36" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32913 } } }
I20260929 17:53:36.863349 8490 tablet_replica.cc:442] T bbe5590ef0094985a27a4ba2caf10534 P 4e821d34d1844842be7365e97204614e: This TxnStatusTablet replica's current role is: LEADER
I20260929 17:53:36.863605 8492 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260929 17:53:36.863754 8492 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260929 17:53:36.864385 7937 catalog_manager.cc:5495] T bbe5590ef0094985a27a4ba2caf10534 P 4e821d34d1844842be7365e97204614e reported cstate change: term changed from 0 to 1, leader changed from <none> to 4e821d34d1844842be7365e97204614e (127.0.98.2). New cstate: current_term: 1 leader_uuid: "4e821d34d1844842be7365e97204614e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ece5590526f94058b07afe5bb610ccfa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e821d34d1844842be7365e97204614e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43829 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8be608a8c8b43e6a7063022a8a7da36" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32913 } health_report { overall_health: UNKNOWN } } }
I20260929 17:53:36.882409 8448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:36.878792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46004 (local address 127.0.98.2:43829)
0929 17:53:36.878897 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:36.878900 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:36.878985 (+ 85us) server_negotiation.cc:407] Connection header received
0929 17:53:36.879154 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:36.879157 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:36.879207 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:36.879288 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 17:53:36.879491 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:36.879976 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:36.881138 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:36.881349 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:36.881423 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:36.881685 (+ 262us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 17:53:36.882087 (+ 402us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 17:53:36.882169 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0929 17:53:36.882256 (+ 87us) server_negotiation.cc:299] Negotiation successful
0929 17:53:36.882302 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260929 17:53:36.882464 8484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:36.878694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43829 (local address 127.0.0.1:46004)
0929 17:53:36.878921 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:36.878935 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 17:53:36.879039 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:53:36.879306 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:53:36.879309 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:53:36.879316 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 17:53:36.879418 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:36.879424 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:36.880057 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:36.880059 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:36.881036 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:36.881046 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:36.881437 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:36.881440 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:36.881523 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:36.881572 (+ 49us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 17:53:36.882165 (+ 593us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 17:53:36.882181 (+ 16us) client_negotiation.cc:769] Sending connection context
0929 17:53:36.882248 (+ 67us) client_negotiation.cc:240] Negotiation successful
0929 17:53:36.882328 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136}
I20260929 17:53:36.885454 8095 raft_consensus.cc:1261] T bbe5590ef0094985a27a4ba2caf10534 P ece5590526f94058b07afe5bb610ccfa [term 1 FOLLOWER]: Refusing update from remote peer 4e821d34d1844842be7365e97204614e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 17:53:36.885643 8375 raft_consensus.cc:1261] T bbe5590ef0094985a27a4ba2caf10534 P c8be608a8c8b43e6a7063022a8a7da36 [term 1 FOLLOWER]: Refusing update from remote peer 4e821d34d1844842be7365e97204614e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 17:53:36.885874 8456 consensus_queue.cc:1035] T bbe5590ef0094985a27a4ba2caf10534 P 4e821d34d1844842be7365e97204614e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ece5590526f94058b07afe5bb610ccfa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 17:53:36.886089 8490 consensus_queue.cc:1035] T bbe5590ef0094985a27a4ba2caf10534 P 4e821d34d1844842be7365e97204614e [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8be608a8c8b43e6a7063022a8a7da36" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 17:53:36.887475 8457 tablet_replica.cc:439] T bbe5590ef0094985a27a4ba2caf10534 P ece5590526f94058b07afe5bb610ccfa: TxnStatusTablet state changed. Reason: New leader 4e821d34d1844842be7365e97204614e. Latest consensus state: current_term: 1 leader_uuid: "4e821d34d1844842be7365e97204614e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ece5590526f94058b07afe5bb610ccfa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40515 } } peers { permanent_uuid: "4e821d34d1844842be7365e97204614e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43829 } } peers { permanent_uuid: "c8be608a8c8b43e6a7063022a8a7da36" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32913 } } }
I20260929 17:53:36.887581 8457 tablet_replica.cc:442] T bbe5590ef0094985a27a4ba2caf10534 P ece5590526f94058b07afe5bb610ccfa: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 17:53:36.887521 8469 tablet_replica.cc:439] T bbe5590ef0094985a27a4ba2caf10534 P c8be608a8c8b43e6a7063022a8a7da36: TxnStatusTablet state changed. Reason: New leader 4e821d34d1844842be7365e97204614e. Latest consensus state: current_term: 1 leader_uuid: "4e821d34d1844842be7365e97204614e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ece5590526f94058b07afe5bb610ccfa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40515 } } peers { permanent_uuid: "4e821d34d1844842be7365e97204614e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43829 } } peers { permanent_uuid: "c8be608a8c8b43e6a7063022a8a7da36" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32913 } } }
I20260929 17:53:36.887614 8469 tablet_replica.cc:442] T bbe5590ef0094985a27a4ba2caf10534 P c8be608a8c8b43e6a7063022a8a7da36: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 17:53:36.889190 8490 tablet_replica.cc:439] T bbe5590ef0094985a27a4ba2caf10534 P 4e821d34d1844842be7365e97204614e: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4e821d34d1844842be7365e97204614e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ece5590526f94058b07afe5bb610ccfa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40515 } } peers { permanent_uuid: "4e821d34d1844842be7365e97204614e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43829 } } peers { permanent_uuid: "c8be608a8c8b43e6a7063022a8a7da36" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32913 } } }
I20260929 17:53:36.889286 8490 tablet_replica.cc:442] T bbe5590ef0094985a27a4ba2caf10534 P 4e821d34d1844842be7365e97204614e: This TxnStatusTablet replica's current role is: LEADER
I20260929 17:53:36.889369 8464 mvcc.cc:205] Tried to move back new op lower bound from 7334725291557937152 to 7334725291469475840. Current Snapshot: MvccSnapshot[applied={T|T < 7334725291557937152}]
I20260929 17:53:36.889557 8462 mvcc.cc:205] Tried to move back new op lower bound from 7334725291557937152 to 7334725291469475840. Current Snapshot: MvccSnapshot[applied={T|T < 7334725291557937152}]
I20260929 17:53:36.889701 8469 tablet_replica.cc:439] T bbe5590ef0094985a27a4ba2caf10534 P c8be608a8c8b43e6a7063022a8a7da36: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4e821d34d1844842be7365e97204614e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ece5590526f94058b07afe5bb610ccfa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40515 } } peers { permanent_uuid: "4e821d34d1844842be7365e97204614e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43829 } } peers { permanent_uuid: "c8be608a8c8b43e6a7063022a8a7da36" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32913 } } }
I20260929 17:53:36.889801 8469 tablet_replica.cc:442] T bbe5590ef0094985a27a4ba2caf10534 P c8be608a8c8b43e6a7063022a8a7da36: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 17:53:36.890707 8456 tablet_replica.cc:439] T bbe5590ef0094985a27a4ba2caf10534 P 4e821d34d1844842be7365e97204614e: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4e821d34d1844842be7365e97204614e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ece5590526f94058b07afe5bb610ccfa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40515 } } peers { permanent_uuid: "4e821d34d1844842be7365e97204614e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43829 } } peers { permanent_uuid: "c8be608a8c8b43e6a7063022a8a7da36" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32913 } } }
I20260929 17:53:36.890823 8456 tablet_replica.cc:442] T bbe5590ef0094985a27a4ba2caf10534 P 4e821d34d1844842be7365e97204614e: This TxnStatusTablet replica's current role is: LEADER
I20260929 17:53:36.891477 8457 tablet_replica.cc:439] T bbe5590ef0094985a27a4ba2caf10534 P ece5590526f94058b07afe5bb610ccfa: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4e821d34d1844842be7365e97204614e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ece5590526f94058b07afe5bb610ccfa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40515 } } peers { permanent_uuid: "4e821d34d1844842be7365e97204614e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43829 } } peers { permanent_uuid: "c8be608a8c8b43e6a7063022a8a7da36" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32913 } } }
I20260929 17:53:36.891571 8457 tablet_replica.cc:442] T bbe5590ef0094985a27a4ba2caf10534 P ece5590526f94058b07afe5bb610ccfa: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 17:53:36.905607 8500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:36.902209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43829 (local address 127.0.98.3:54059)
0929 17:53:36.902520 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:36.902531 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 17:53:36.902619 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:53:36.902822 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:53:36.902824 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:53:36.902830 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 17:53:36.902916 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:36.902920 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:36.903864 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:36.903866 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:36.904470 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:36.904476 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:36.904981 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:36.904984 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:36.905082 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:36.905125 (+ 43us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 17:53:36.905382 (+ 257us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 17:53:36.905386 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 17:53:36.905427 (+ 41us) client_negotiation.cc:240] Negotiation successful
0929 17:53:36.905493 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":78,"threads_started":1}
I20260929 17:53:36.905616 8448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:36.902288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54059 (local address 127.0.98.2:43829)
0929 17:53:36.902391 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:36.902394 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:36.902603 (+ 209us) server_negotiation.cc:407] Connection header received
0929 17:53:36.902691 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:36.902694 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:36.902743 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:36.902819 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 17:53:36.902995 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:36.903780 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:36.904560 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:36.904888 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:36.904987 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:36.905236 (+ 249us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 17:53:36.905314 (+ 78us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 17:53:36.905381 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0929 17:53:36.905438 (+ 57us) server_negotiation.cc:299] Negotiation successful
0929 17:53:36.905487 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260929 17:53:36.911407 8095 raft_consensus.cc:1261] T dd520d47442846f183e4292f0df97d64 P ece5590526f94058b07afe5bb610ccfa [term 1 FOLLOWER]: Refusing update from remote peer c8be608a8c8b43e6a7063022a8a7da36: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 17:53:36.911443 8235 raft_consensus.cc:1261] T dd520d47442846f183e4292f0df97d64 P 4e821d34d1844842be7365e97204614e [term 1 FOLLOWER]: Refusing update from remote peer c8be608a8c8b43e6a7063022a8a7da36: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 17:53:36.911754 8469 consensus_queue.cc:1035] T dd520d47442846f183e4292f0df97d64 P c8be608a8c8b43e6a7063022a8a7da36 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ece5590526f94058b07afe5bb610ccfa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 17:53:36.911974 8459 consensus_queue.cc:1035] T dd520d47442846f183e4292f0df97d64 P c8be608a8c8b43e6a7063022a8a7da36 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e821d34d1844842be7365e97204614e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 17:53:36.932374 8444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:36.928759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49365 (local address 127.0.98.3:32913)
0929 17:53:36.928886 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:36.928891 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:36.929097 (+ 206us) server_negotiation.cc:407] Connection header received
0929 17:53:36.929234 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:36.929237 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:36.929286 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:36.929344 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 17:53:36.929625 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:36.930137 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:36.931138 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:36.931449 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:36.931538 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:36.931855 (+ 317us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 17:53:36.931937 (+ 82us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 17:53:36.932026 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0929 17:53:36.932160 (+ 134us) server_negotiation.cc:299] Negotiation successful
0929 17:53:36.932244 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260929 17:53:36.932379 8504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:36.928701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:32913 (local address 127.0.98.2:49365)
0929 17:53:36.929023 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:36.929039 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 17:53:36.929143 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:53:36.929357 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:53:36.929360 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:53:36.929367 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 17:53:36.929487 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:36.929510 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:36.930245 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:36.930247 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:36.931034 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:36.931044 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:36.931561 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:36.931565 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:36.931673 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:36.931731 (+ 58us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 17:53:36.932067 (+ 336us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 17:53:36.932073 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 17:53:36.932129 (+ 56us) client_negotiation.cc:240] Negotiation successful
0929 17:53:36.932209 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":104,"threads_started":1}
I20260929 17:53:36.965493 7936 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51932:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 17:53:36.965675 7936 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51932:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 17:53:36.967865 7936 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1fb78915f2ed49e5b241da7350b145ae: Sending DeleteTablet for 3 replicas of tablet dd520d47442846f183e4292f0df97d64
I20260929 17:53:36.968477 8075 tablet_service.cc:1514] Processing DeleteTablet for tablet dd520d47442846f183e4292f0df97d64 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 17:53:36 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:32840
I20260929 17:53:36.968640 8355 tablet_service.cc:1514] Processing DeleteTablet for tablet dd520d47442846f183e4292f0df97d64 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 17:53:36 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43864
I20260929 17:53:36.968693 8472 tablet_replica.cc:330] T dd520d47442846f183e4292f0df97d64 P ece5590526f94058b07afe5bb610ccfa: stopping tablet replica
I20260929 17:53:36.968823 8472 raft_consensus.cc:2229] T dd520d47442846f183e4292f0df97d64 P ece5590526f94058b07afe5bb610ccfa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 17:53:36.968827 8474 tablet_replica.cc:330] T dd520d47442846f183e4292f0df97d64 P c8be608a8c8b43e6a7063022a8a7da36: stopping tablet replica
I20260929 17:53:36.968950 8472 raft_consensus.cc:2258] T dd520d47442846f183e4292f0df97d64 P ece5590526f94058b07afe5bb610ccfa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 17:53:36.968947 8474 raft_consensus.cc:2229] T dd520d47442846f183e4292f0df97d64 P c8be608a8c8b43e6a7063022a8a7da36 [term 1 LEADER]: Raft consensus shutting down.
I20260929 17:53:36.969147 8215 tablet_service.cc:1514] Processing DeleteTablet for tablet dd520d47442846f183e4292f0df97d64 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 17:53:36 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45988
I20260929 17:53:36.969163 8474 raft_consensus.cc:2258] T dd520d47442846f183e4292f0df97d64 P c8be608a8c8b43e6a7063022a8a7da36 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 17:53:36.969326 8473 tablet_replica.cc:330] T dd520d47442846f183e4292f0df97d64 P 4e821d34d1844842be7365e97204614e: stopping tablet replica
I20260929 17:53:36.969434 8473 raft_consensus.cc:2229] T dd520d47442846f183e4292f0df97d64 P 4e821d34d1844842be7365e97204614e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 17:53:36.969539 8473 raft_consensus.cc:2258] T dd520d47442846f183e4292f0df97d64 P 4e821d34d1844842be7365e97204614e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 17:53:36.970026 8474 ts_tablet_manager.cc:1905] T dd520d47442846f183e4292f0df97d64 P c8be608a8c8b43e6a7063022a8a7da36: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 17:53:36.970180 8472 ts_tablet_manager.cc:1905] T dd520d47442846f183e4292f0df97d64 P ece5590526f94058b07afe5bb610ccfa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 17:53:36.973361 8472 ts_tablet_manager.cc:1918] T dd520d47442846f183e4292f0df97d64 P ece5590526f94058b07afe5bb610ccfa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 17:53:36.973456 8472 log.cc:1196] T dd520d47442846f183e4292f0df97d64 P ece5590526f94058b07afe5bb610ccfa: Deleting WAL directory at /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704375975685-392-0/minicluster-data/ts-0/wal/wals/dd520d47442846f183e4292f0df97d64
I20260929 17:53:36.973696 8472 ts_tablet_manager.cc:1939] T dd520d47442846f183e4292f0df97d64 P ece5590526f94058b07afe5bb610ccfa: Deleting consensus metadata
I20260929 17:53:36.974094 8474 ts_tablet_manager.cc:1918] T dd520d47442846f183e4292f0df97d64 P c8be608a8c8b43e6a7063022a8a7da36: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 17:53:36.974191 8474 log.cc:1196] T dd520d47442846f183e4292f0df97d64 P c8be608a8c8b43e6a7063022a8a7da36: Deleting WAL directory at /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704375975685-392-0/minicluster-data/ts-2/wal/wals/dd520d47442846f183e4292f0df97d64
I20260929 17:53:36.974436 8474 ts_tablet_manager.cc:1939] T dd520d47442846f183e4292f0df97d64 P c8be608a8c8b43e6a7063022a8a7da36: Deleting consensus metadata
I20260929 17:53:36.974531 7922 catalog_manager.cc:4841] TS ece5590526f94058b07afe5bb610ccfa (127.0.98.1:40515): tablet dd520d47442846f183e4292f0df97d64 (table test-table [id=420b3c3ab44a4bf7aa830edb0fc4c7c5]) successfully deleted
I20260929 17:53:36.975178 7925 catalog_manager.cc:4841] TS c8be608a8c8b43e6a7063022a8a7da36 (127.0.98.3:32913): tablet dd520d47442846f183e4292f0df97d64 (table test-table [id=420b3c3ab44a4bf7aa830edb0fc4c7c5]) successfully deleted
I20260929 17:53:36.975773 8473 ts_tablet_manager.cc:1905] T dd520d47442846f183e4292f0df97d64 P 4e821d34d1844842be7365e97204614e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 17:53:36.978044 8473 ts_tablet_manager.cc:1918] T dd520d47442846f183e4292f0df97d64 P 4e821d34d1844842be7365e97204614e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 17:53:36.978116 8473 log.cc:1196] T dd520d47442846f183e4292f0df97d64 P 4e821d34d1844842be7365e97204614e: Deleting WAL directory at /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704375975685-392-0/minicluster-data/ts-1/wal/wals/dd520d47442846f183e4292f0df97d64
I20260929 17:53:36.978295 8473 ts_tablet_manager.cc:1939] T dd520d47442846f183e4292f0df97d64 P 4e821d34d1844842be7365e97204614e: Deleting consensus metadata
I20260929 17:53:36.979452 7925 catalog_manager.cc:4841] TS 4e821d34d1844842be7365e97204614e (127.0.98.2:43829): tablet dd520d47442846f183e4292f0df97d64 (table test-table [id=420b3c3ab44a4bf7aa830edb0fc4c7c5]) successfully deleted
Sep 29 17:53:36 dist-test-slave-53nh krb5kdc[7893](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704416, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 17:53:36 dist-test-slave-53nh krb5kdc[7893](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 17:53:36 dist-test-slave-53nh krb5kdc[7893](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260929 17:53:36.999598 7987 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:53:36.992456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51948 (local address 127.0.98.62:44965)
0929 17:53:36.992585 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:36.992598 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:36.992715 (+ 117us) server_negotiation.cc:407] Connection header received
0929 17:53:36.992866 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:36.992869 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:36.992910 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:36.992978 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:53:36.993950 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:36.994464 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:36.995649 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:36.995860 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:36.995929 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:36.999461 (+ 3532us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51948: BlockingRecv error: recv got EOF from 127.0.0.1:51948 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 29 17:53:37 dist-test-slave-53nh krb5kdc[7893](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704416, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260929 17:53:37.010008 7987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:37.001097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51962 (local address 127.0.98.62:44965)
0929 17:53:37.001223 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:37.001228 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:37.001386 (+ 158us) server_negotiation.cc:407] Connection header received
0929 17:53:37.001498 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:37.001500 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:37.001544 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:37.001611 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:53:37.002325 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:37.002821 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:37.003991 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:37.004189 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:37.004257 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:37.006843 (+ 2586us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:53:37.006847 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:53:37.006850 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:53:37.006880 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:53:37.008479 (+ 1599us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:53:37.008872 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:53:37.008876 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:53:37.008877 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:53:37.008968 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:53:37.009230 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:53:37.009233 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:53:37.009235 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:53:37.009440 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:53:37.009559 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0929 17:53:37.009729 (+ 170us) server_negotiation.cc:299] Negotiation successful
0929 17:53:37.009844 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 17:53:37.012600 7936 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51962:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 17:53:37.013139 7936 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 17:53:37.017922 8215 tablet_service.cc:1467] Processing CreateTablet for tablet dff4311a9d334ac4bf49546c254466db (DEFAULT_TABLE table=test-table [id=da7584a275ab45858e723a99f0744836]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 17:53:37.017922 8355 tablet_service.cc:1467] Processing CreateTablet for tablet dff4311a9d334ac4bf49546c254466db (DEFAULT_TABLE table=test-table [id=da7584a275ab45858e723a99f0744836]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 17:53:37.017922 8075 tablet_service.cc:1467] Processing CreateTablet for tablet dff4311a9d334ac4bf49546c254466db (DEFAULT_TABLE table=test-table [id=da7584a275ab45858e723a99f0744836]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 17:53:37.018217 8075 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dff4311a9d334ac4bf49546c254466db. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:53:37.018216 8355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dff4311a9d334ac4bf49546c254466db. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:53:37.018236 8215 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dff4311a9d334ac4bf49546c254466db. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:53:37.020597 8449 tablet_bootstrap.cc:492] T dff4311a9d334ac4bf49546c254466db P 4e821d34d1844842be7365e97204614e: Bootstrap starting.
I20260929 17:53:37.021421 8449 tablet_bootstrap.cc:654] T dff4311a9d334ac4bf49546c254466db P 4e821d34d1844842be7365e97204614e: Neither blocks nor log segments found. Creating new log.
I20260929 17:53:37.021919 8450 tablet_bootstrap.cc:492] T dff4311a9d334ac4bf49546c254466db P c8be608a8c8b43e6a7063022a8a7da36: Bootstrap starting.
I20260929 17:53:37.022810 8450 tablet_bootstrap.cc:654] T dff4311a9d334ac4bf49546c254466db P c8be608a8c8b43e6a7063022a8a7da36: Neither blocks nor log segments found. Creating new log.
I20260929 17:53:37.024940 8451 tablet_bootstrap.cc:492] T dff4311a9d334ac4bf49546c254466db P ece5590526f94058b07afe5bb610ccfa: Bootstrap starting.
I20260929 17:53:37.025820 8451 tablet_bootstrap.cc:654] T dff4311a9d334ac4bf49546c254466db P ece5590526f94058b07afe5bb610ccfa: Neither blocks nor log segments found. Creating new log.
I20260929 17:53:37.025902 8450 tablet_bootstrap.cc:492] T dff4311a9d334ac4bf49546c254466db P c8be608a8c8b43e6a7063022a8a7da36: No bootstrap required, opened a new log
I20260929 17:53:37.025978 8450 ts_tablet_manager.cc:1397] T dff4311a9d334ac4bf49546c254466db P c8be608a8c8b43e6a7063022a8a7da36: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260929 17:53:37.026392 8450 raft_consensus.cc:348] T dff4311a9d334ac4bf49546c254466db P c8be608a8c8b43e6a7063022a8a7da36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ece5590526f94058b07afe5bb610ccfa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40515 } } peers { permanent_uuid: "4e821d34d1844842be7365e97204614e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43829 } } peers { permanent_uuid: "c8be608a8c8b43e6a7063022a8a7da36" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32913 } }
I20260929 17:53:37.026507 8450 raft_consensus.cc:374] T dff4311a9d334ac4bf49546c254466db P c8be608a8c8b43e6a7063022a8a7da36 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:53:37.026538 8450 raft_consensus.cc:729] T dff4311a9d334ac4bf49546c254466db P c8be608a8c8b43e6a7063022a8a7da36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8be608a8c8b43e6a7063022a8a7da36, State: Initialized, Role: FOLLOWER
I20260929 17:53:37.026655 8450 consensus_queue.cc:260] T dff4311a9d334ac4bf49546c254466db P c8be608a8c8b43e6a7063022a8a7da36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ece5590526f94058b07afe5bb610ccfa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40515 } } peers { permanent_uuid: "4e821d34d1844842be7365e97204614e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43829 } } peers { permanent_uuid: "c8be608a8c8b43e6a7063022a8a7da36" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32913 } }
I20260929 17:53:37.026921 8450 ts_tablet_manager.cc:1428] T dff4311a9d334ac4bf49546c254466db P c8be608a8c8b43e6a7063022a8a7da36: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 17:53:37.027918 8451 tablet_bootstrap.cc:492] T dff4311a9d334ac4bf49546c254466db P ece5590526f94058b07afe5bb610ccfa: No bootstrap required, opened a new log
I20260929 17:53:37.027998 8451 ts_tablet_manager.cc:1397] T dff4311a9d334ac4bf49546c254466db P ece5590526f94058b07afe5bb610ccfa: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260929 17:53:37.028437 8451 raft_consensus.cc:348] T dff4311a9d334ac4bf49546c254466db P ece5590526f94058b07afe5bb610ccfa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ece5590526f94058b07afe5bb610ccfa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40515 } } peers { permanent_uuid: "4e821d34d1844842be7365e97204614e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43829 } } peers { permanent_uuid: "c8be608a8c8b43e6a7063022a8a7da36" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32913 } }
I20260929 17:53:37.028528 8451 raft_consensus.cc:374] T dff4311a9d334ac4bf49546c254466db P ece5590526f94058b07afe5bb610ccfa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:53:37.028532 8449 tablet_bootstrap.cc:492] T dff4311a9d334ac4bf49546c254466db P 4e821d34d1844842be7365e97204614e: No bootstrap required, opened a new log
I20260929 17:53:37.028558 8451 raft_consensus.cc:729] T dff4311a9d334ac4bf49546c254466db P ece5590526f94058b07afe5bb610ccfa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ece5590526f94058b07afe5bb610ccfa, State: Initialized, Role: FOLLOWER
I20260929 17:53:37.028612 8449 ts_tablet_manager.cc:1397] T dff4311a9d334ac4bf49546c254466db P 4e821d34d1844842be7365e97204614e: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.000s
I20260929 17:53:37.029019 8449 raft_consensus.cc:348] T dff4311a9d334ac4bf49546c254466db P 4e821d34d1844842be7365e97204614e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ece5590526f94058b07afe5bb610ccfa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40515 } } peers { permanent_uuid: "4e821d34d1844842be7365e97204614e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43829 } } peers { permanent_uuid: "c8be608a8c8b43e6a7063022a8a7da36" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32913 } }
I20260929 17:53:37.028680 8451 consensus_queue.cc:260] T dff4311a9d334ac4bf49546c254466db P ece5590526f94058b07afe5bb610ccfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ece5590526f94058b07afe5bb610ccfa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40515 } } peers { permanent_uuid: "4e821d34d1844842be7365e97204614e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43829 } } peers { permanent_uuid: "c8be608a8c8b43e6a7063022a8a7da36" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32913 } }
I20260929 17:53:37.029130 8449 raft_consensus.cc:374] T dff4311a9d334ac4bf49546c254466db P 4e821d34d1844842be7365e97204614e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:53:37.029157 8449 raft_consensus.cc:729] T dff4311a9d334ac4bf49546c254466db P 4e821d34d1844842be7365e97204614e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e821d34d1844842be7365e97204614e, State: Initialized, Role: FOLLOWER
I20260929 17:53:37.029273 8449 consensus_queue.cc:260] T dff4311a9d334ac4bf49546c254466db P 4e821d34d1844842be7365e97204614e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ece5590526f94058b07afe5bb610ccfa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40515 } } peers { permanent_uuid: "4e821d34d1844842be7365e97204614e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43829 } } peers { permanent_uuid: "c8be608a8c8b43e6a7063022a8a7da36" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32913 } }
I20260929 17:53:37.029325 8451 ts_tablet_manager.cc:1428] T dff4311a9d334ac4bf49546c254466db P ece5590526f94058b07afe5bb610ccfa: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20260929 17:53:37.029500 8449 ts_tablet_manager.cc:1428] T dff4311a9d334ac4bf49546c254466db P 4e821d34d1844842be7365e97204614e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 17:53:37.075788 8457 raft_consensus.cc:482] T dff4311a9d334ac4bf49546c254466db P ece5590526f94058b07afe5bb610ccfa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 17:53:37.075904 8457 raft_consensus.cc:504] T dff4311a9d334ac4bf49546c254466db P ece5590526f94058b07afe5bb610ccfa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ece5590526f94058b07afe5bb610ccfa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40515 } } peers { permanent_uuid: "4e821d34d1844842be7365e97204614e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43829 } } peers { permanent_uuid: "c8be608a8c8b43e6a7063022a8a7da36" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32913 } }
I20260929 17:53:37.076915 8457 leader_election.cc:288] T dff4311a9d334ac4bf49546c254466db P ece5590526f94058b07afe5bb610ccfa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4e821d34d1844842be7365e97204614e (127.0.98.2:43829), c8be608a8c8b43e6a7063022a8a7da36 (127.0.98.3:32913)
I20260929 17:53:37.080547 8448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:37.077249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55557 (local address 127.0.98.2:43829)
0929 17:53:37.077411 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:37.077415 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:37.077520 (+ 105us) server_negotiation.cc:407] Connection header received
0929 17:53:37.077675 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:37.077679 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:37.077733 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:37.077805 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:53:37.078069 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:37.078558 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:37.079622 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:37.080011 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:37.080103 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:37.080170 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0929 17:53:37.080363 (+ 193us) server_negotiation.cc:299] Negotiation successful
0929 17:53:37.080425 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 17:53:37.080569 8521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:37.077171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43829 (local address 127.0.98.1:55557)
0929 17:53:37.077445 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:37.077457 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 17:53:37.077564 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:53:37.077829 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:53:37.077832 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:53:37.077850 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:53:37.077971 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:37.077975 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:37.078685 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:37.078688 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:37.079504 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:37.079513 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:37.080156 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:37.080160 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:37.080250 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:37.080271 (+ 21us) client_negotiation.cc:769] Sending connection context
0929 17:53:37.080337 (+ 66us) client_negotiation.cc:240] Negotiation successful
0929 17:53:37.080409 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":100,"threads_started":1}
I20260929 17:53:37.080754 8444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:37.077667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45287 (local address 127.0.98.3:32913)
0929 17:53:37.077787 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:37.077792 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:37.077935 (+ 143us) server_negotiation.cc:407] Connection header received
0929 17:53:37.078034 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:37.078037 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:37.078081 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:37.078140 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:53:37.078391 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:37.078876 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:37.079853 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:37.080232 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:37.080318 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:37.080374 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0929 17:53:37.080551 (+ 177us) server_negotiation.cc:299] Negotiation successful
0929 17:53:37.080619 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 17:53:37.080961 8235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dff4311a9d334ac4bf49546c254466db" candidate_uuid: "ece5590526f94058b07afe5bb610ccfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e821d34d1844842be7365e97204614e" is_pre_election: true
I20260929 17:53:37.081107 8522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:37.077542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:32913 (local address 127.0.98.1:45287)
0929 17:53:37.077856 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:37.077867 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 17:53:37.077952 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:53:37.078158 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:53:37.078161 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:53:37.078183 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:53:37.078296 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:37.078301 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:37.078970 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:37.078972 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:37.079738 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:37.079746 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:37.080333 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:37.080337 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:37.080440 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:37.080461 (+ 21us) client_negotiation.cc:769] Sending connection context
0929 17:53:37.080523 (+ 62us) client_negotiation.cc:240] Negotiation successful
0929 17:53:37.080588 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"mutex_wait_us":1,"thread_start_us":103,"threads_started":1}
I20260929 17:53:37.081138 8235 raft_consensus.cc:2454] T dff4311a9d334ac4bf49546c254466db P 4e821d34d1844842be7365e97204614e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ece5590526f94058b07afe5bb610ccfa in term 0.
I20260929 17:53:37.081475 8029 leader_election.cc:302] T dff4311a9d334ac4bf49546c254466db P ece5590526f94058b07afe5bb610ccfa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e821d34d1844842be7365e97204614e, ece5590526f94058b07afe5bb610ccfa; no voters:
I20260929 17:53:37.081498 8375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dff4311a9d334ac4bf49546c254466db" candidate_uuid: "ece5590526f94058b07afe5bb610ccfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8be608a8c8b43e6a7063022a8a7da36" is_pre_election: true
I20260929 17:53:37.081610 8375 raft_consensus.cc:2454] T dff4311a9d334ac4bf49546c254466db P c8be608a8c8b43e6a7063022a8a7da36 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ece5590526f94058b07afe5bb610ccfa in term 0.
I20260929 17:53:37.081713 8457 raft_consensus.cc:2781] T dff4311a9d334ac4bf49546c254466db P ece5590526f94058b07afe5bb610ccfa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 17:53:37.081763 8457 raft_consensus.cc:482] T dff4311a9d334ac4bf49546c254466db P ece5590526f94058b07afe5bb610ccfa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 17:53:37.081794 8457 raft_consensus.cc:3037] T dff4311a9d334ac4bf49546c254466db P ece5590526f94058b07afe5bb610ccfa [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:53:37.082692 8457 raft_consensus.cc:504] T dff4311a9d334ac4bf49546c254466db P ece5590526f94058b07afe5bb610ccfa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ece5590526f94058b07afe5bb610ccfa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40515 } } peers { permanent_uuid: "4e821d34d1844842be7365e97204614e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43829 } } peers { permanent_uuid: "c8be608a8c8b43e6a7063022a8a7da36" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32913 } }
I20260929 17:53:37.082976 8457 leader_election.cc:288] T dff4311a9d334ac4bf49546c254466db P ece5590526f94058b07afe5bb610ccfa [CANDIDATE]: Term 1 election: Requested vote from peers 4e821d34d1844842be7365e97204614e (127.0.98.2:43829), c8be608a8c8b43e6a7063022a8a7da36 (127.0.98.3:32913)
I20260929 17:53:37.083370 8235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dff4311a9d334ac4bf49546c254466db" candidate_uuid: "ece5590526f94058b07afe5bb610ccfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e821d34d1844842be7365e97204614e"
I20260929 17:53:37.083412 8375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dff4311a9d334ac4bf49546c254466db" candidate_uuid: "ece5590526f94058b07afe5bb610ccfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8be608a8c8b43e6a7063022a8a7da36"
I20260929 17:53:37.083495 8235 raft_consensus.cc:3037] T dff4311a9d334ac4bf49546c254466db P 4e821d34d1844842be7365e97204614e [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:53:37.083514 8375 raft_consensus.cc:3037] T dff4311a9d334ac4bf49546c254466db P c8be608a8c8b43e6a7063022a8a7da36 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:53:37.084547 8375 raft_consensus.cc:2454] T dff4311a9d334ac4bf49546c254466db P c8be608a8c8b43e6a7063022a8a7da36 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ece5590526f94058b07afe5bb610ccfa in term 1.
I20260929 17:53:37.084682 8235 raft_consensus.cc:2454] T dff4311a9d334ac4bf49546c254466db P 4e821d34d1844842be7365e97204614e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ece5590526f94058b07afe5bb610ccfa in term 1.
I20260929 17:53:37.084807 8029 leader_election.cc:302] T dff4311a9d334ac4bf49546c254466db P ece5590526f94058b07afe5bb610ccfa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c8be608a8c8b43e6a7063022a8a7da36, ece5590526f94058b07afe5bb610ccfa; no voters:
I20260929 17:53:37.084981 8457 raft_consensus.cc:2781] T dff4311a9d334ac4bf49546c254466db P ece5590526f94058b07afe5bb610ccfa [term 1 FOLLOWER]: Leader election won for term 1
I20260929 17:53:37.085228 8457 raft_consensus.cc:686] T dff4311a9d334ac4bf49546c254466db P ece5590526f94058b07afe5bb610ccfa [term 1 LEADER]: Becoming Leader. State: Replica: ece5590526f94058b07afe5bb610ccfa, State: Running, Role: LEADER
I20260929 17:53:37.085471 8457 consensus_queue.cc:237] T dff4311a9d334ac4bf49546c254466db P ece5590526f94058b07afe5bb610ccfa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ece5590526f94058b07afe5bb610ccfa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40515 } } peers { permanent_uuid: "4e821d34d1844842be7365e97204614e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43829 } } peers { permanent_uuid: "c8be608a8c8b43e6a7063022a8a7da36" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32913 } }
I20260929 17:53:37.087639 7938 catalog_manager.cc:5495] T dff4311a9d334ac4bf49546c254466db P ece5590526f94058b07afe5bb610ccfa reported cstate change: term changed from 0 to 1, leader changed from <none> to ece5590526f94058b07afe5bb610ccfa (127.0.98.1). New cstate: current_term: 1 leader_uuid: "ece5590526f94058b07afe5bb610ccfa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ece5590526f94058b07afe5bb610ccfa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40515 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4e821d34d1844842be7365e97204614e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43829 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8be608a8c8b43e6a7063022a8a7da36" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32913 } health_report { overall_health: UNKNOWN } } }
I20260929 17:53:37.109683 8447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:37.106293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32856 (local address 127.0.98.1:40515)
0929 17:53:37.106418 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:37.106421 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:37.106434 (+ 13us) server_negotiation.cc:407] Connection header received
0929 17:53:37.106541 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:37.106543 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:37.106596 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:37.106687 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 17:53:37.106872 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:37.107370 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:37.108327 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:37.108549 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:37.108629 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:37.108839 (+ 210us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 17:53:37.109342 (+ 503us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 17:53:37.109430 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0929 17:53:37.109484 (+ 54us) server_negotiation.cc:299] Negotiation successful
0929 17:53:37.109537 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 17:53:37.112608 8375 raft_consensus.cc:1261] T dff4311a9d334ac4bf49546c254466db P c8be608a8c8b43e6a7063022a8a7da36 [term 1 FOLLOWER]: Refusing update from remote peer ece5590526f94058b07afe5bb610ccfa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 17:53:37.112658 8235 raft_consensus.cc:1261] T dff4311a9d334ac4bf49546c254466db P 4e821d34d1844842be7365e97204614e [term 1 FOLLOWER]: Refusing update from remote peer ece5590526f94058b07afe5bb610ccfa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 17:53:37.113194 8457 consensus_queue.cc:1035] T dff4311a9d334ac4bf49546c254466db P ece5590526f94058b07afe5bb610ccfa [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e821d34d1844842be7365e97204614e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 17:53:37.113407 8523 consensus_queue.cc:1035] T dff4311a9d334ac4bf49546c254466db P ece5590526f94058b07afe5bb610ccfa [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8be608a8c8b43e6a7063022a8a7da36" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 17:53:37.124060 7937 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51962:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 17:53:37.124213 7937 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51962:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 17:53:37.126236 7937 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1fb78915f2ed49e5b241da7350b145ae: Sending DeleteTablet for 3 replicas of tablet dff4311a9d334ac4bf49546c254466db
I20260929 17:53:37.126760 8075 tablet_service.cc:1514] Processing DeleteTablet for tablet dff4311a9d334ac4bf49546c254466db with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 17:53:37 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:32840
I20260929 17:53:37.126947 8472 tablet_replica.cc:330] T dff4311a9d334ac4bf49546c254466db P ece5590526f94058b07afe5bb610ccfa: stopping tablet replica
I20260929 17:53:37.126995 8355 tablet_service.cc:1514] Processing DeleteTablet for tablet dff4311a9d334ac4bf49546c254466db with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 17:53:37 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43864
I20260929 17:53:37.127004 8215 tablet_service.cc:1514] Processing DeleteTablet for tablet dff4311a9d334ac4bf49546c254466db with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 17:53:37 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45988
I20260929 17:53:37.127074 8472 raft_consensus.cc:2229] T dff4311a9d334ac4bf49546c254466db P ece5590526f94058b07afe5bb610ccfa [term 1 LEADER]: Raft consensus shutting down.
I20260929 17:53:37.127228 8474 tablet_replica.cc:330] T dff4311a9d334ac4bf49546c254466db P c8be608a8c8b43e6a7063022a8a7da36: stopping tablet replica
I20260929 17:53:37.127346 8474 raft_consensus.cc:2229] T dff4311a9d334ac4bf49546c254466db P c8be608a8c8b43e6a7063022a8a7da36 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 17:53:37.127345 8472 raft_consensus.cc:2258] T dff4311a9d334ac4bf49546c254466db P ece5590526f94058b07afe5bb610ccfa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 17:53:37.127465 8474 raft_consensus.cc:2258] T dff4311a9d334ac4bf49546c254466db P c8be608a8c8b43e6a7063022a8a7da36 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 17:53:37.127622 8473 tablet_replica.cc:330] T dff4311a9d334ac4bf49546c254466db P 4e821d34d1844842be7365e97204614e: stopping tablet replica
I20260929 17:53:37.127732 8473 raft_consensus.cc:2229] T dff4311a9d334ac4bf49546c254466db P 4e821d34d1844842be7365e97204614e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 17:53:37.127853 8473 raft_consensus.cc:2258] T dff4311a9d334ac4bf49546c254466db P 4e821d34d1844842be7365e97204614e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 17:53:37.128013 8472 ts_tablet_manager.cc:1905] T dff4311a9d334ac4bf49546c254466db P ece5590526f94058b07afe5bb610ccfa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 17:53:37.128532 8473 ts_tablet_manager.cc:1905] T dff4311a9d334ac4bf49546c254466db P 4e821d34d1844842be7365e97204614e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 17:53:37.130344 8473 ts_tablet_manager.cc:1918] T dff4311a9d334ac4bf49546c254466db P 4e821d34d1844842be7365e97204614e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 17:53:37.130427 8473 log.cc:1196] T dff4311a9d334ac4bf49546c254466db P 4e821d34d1844842be7365e97204614e: Deleting WAL directory at /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704375975685-392-0/minicluster-data/ts-1/wal/wals/dff4311a9d334ac4bf49546c254466db
I20260929 17:53:37.130527 8472 ts_tablet_manager.cc:1918] T dff4311a9d334ac4bf49546c254466db P ece5590526f94058b07afe5bb610ccfa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 17:53:37.130594 8473 ts_tablet_manager.cc:1939] T dff4311a9d334ac4bf49546c254466db P 4e821d34d1844842be7365e97204614e: Deleting consensus metadata
I20260929 17:53:37.130595 8472 log.cc:1196] T dff4311a9d334ac4bf49546c254466db P ece5590526f94058b07afe5bb610ccfa: Deleting WAL directory at /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704375975685-392-0/minicluster-data/ts-0/wal/wals/dff4311a9d334ac4bf49546c254466db
I20260929 17:53:37.130826 8472 ts_tablet_manager.cc:1939] T dff4311a9d334ac4bf49546c254466db P ece5590526f94058b07afe5bb610ccfa: Deleting consensus metadata
I20260929 17:53:37.131152 7925 catalog_manager.cc:4841] TS 4e821d34d1844842be7365e97204614e (127.0.98.2:43829): tablet dff4311a9d334ac4bf49546c254466db (table test-table [id=da7584a275ab45858e723a99f0744836]) successfully deleted
I20260929 17:53:37.131479 7922 catalog_manager.cc:4841] TS ece5590526f94058b07afe5bb610ccfa (127.0.98.1:40515): tablet dff4311a9d334ac4bf49546c254466db (table test-table [id=da7584a275ab45858e723a99f0744836]) successfully deleted
I20260929 17:53:37.131434 7937 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51962:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 17:53:37.131807 7937 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 17:53:37.132690 8474 ts_tablet_manager.cc:1905] T dff4311a9d334ac4bf49546c254466db P c8be608a8c8b43e6a7063022a8a7da36: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 17:53:37.134900 8474 ts_tablet_manager.cc:1918] T dff4311a9d334ac4bf49546c254466db P c8be608a8c8b43e6a7063022a8a7da36: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 17:53:37.134965 8474 log.cc:1196] T dff4311a9d334ac4bf49546c254466db P c8be608a8c8b43e6a7063022a8a7da36: Deleting WAL directory at /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704375975685-392-0/minicluster-data/ts-2/wal/wals/dff4311a9d334ac4bf49546c254466db
I20260929 17:53:37.135170 8474 ts_tablet_manager.cc:1939] T dff4311a9d334ac4bf49546c254466db P c8be608a8c8b43e6a7063022a8a7da36: Deleting consensus metadata
I20260929 17:53:37.135710 7925 catalog_manager.cc:4841] TS c8be608a8c8b43e6a7063022a8a7da36 (127.0.98.3:32913): tablet dff4311a9d334ac4bf49546c254466db (table test-table [id=da7584a275ab45858e723a99f0744836]) successfully deleted
I20260929 17:53:37.136339 8075 tablet_service.cc:1467] Processing CreateTablet for tablet 9c049e4dfa9041829a79027dec81adc3 (DEFAULT_TABLE table=test-table [id=538ec8f33f874839a31ff2415e711eeb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 17:53:37.136588 8075 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c049e4dfa9041829a79027dec81adc3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:53:37.136677 8355 tablet_service.cc:1467] Processing CreateTablet for tablet 9c049e4dfa9041829a79027dec81adc3 (DEFAULT_TABLE table=test-table [id=538ec8f33f874839a31ff2415e711eeb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 17:53:37.136890 8355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c049e4dfa9041829a79027dec81adc3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:53:37.137238 8215 tablet_service.cc:1467] Processing CreateTablet for tablet 9c049e4dfa9041829a79027dec81adc3 (DEFAULT_TABLE table=test-table [id=538ec8f33f874839a31ff2415e711eeb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 17:53:37.137482 8215 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c049e4dfa9041829a79027dec81adc3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:53:37.138423 8451 tablet_bootstrap.cc:492] T 9c049e4dfa9041829a79027dec81adc3 P ece5590526f94058b07afe5bb610ccfa: Bootstrap starting.
I20260929 17:53:37.139307 8451 tablet_bootstrap.cc:654] T 9c049e4dfa9041829a79027dec81adc3 P ece5590526f94058b07afe5bb610ccfa: Neither blocks nor log segments found. Creating new log.
I20260929 17:53:37.139576 8449 tablet_bootstrap.cc:492] T 9c049e4dfa9041829a79027dec81adc3 P 4e821d34d1844842be7365e97204614e: Bootstrap starting.
I20260929 17:53:37.140312 8449 tablet_bootstrap.cc:654] T 9c049e4dfa9041829a79027dec81adc3 P 4e821d34d1844842be7365e97204614e: Neither blocks nor log segments found. Creating new log.
I20260929 17:53:37.140424 8451 tablet_bootstrap.cc:492] T 9c049e4dfa9041829a79027dec81adc3 P ece5590526f94058b07afe5bb610ccfa: No bootstrap required, opened a new log
I20260929 17:53:37.140492 8451 ts_tablet_manager.cc:1397] T 9c049e4dfa9041829a79027dec81adc3 P ece5590526f94058b07afe5bb610ccfa: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20260929 17:53:37.140892 8451 raft_consensus.cc:348] T 9c049e4dfa9041829a79027dec81adc3 P ece5590526f94058b07afe5bb610ccfa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ece5590526f94058b07afe5bb610ccfa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40515 } } peers { permanent_uuid: "4e821d34d1844842be7365e97204614e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43829 } } peers { permanent_uuid: "c8be608a8c8b43e6a7063022a8a7da36" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32913 } }
I20260929 17:53:37.141036 8451 raft_consensus.cc:374] T 9c049e4dfa9041829a79027dec81adc3 P ece5590526f94058b07afe5bb610ccfa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:53:37.141072 8451 raft_consensus.cc:729] T 9c049e4dfa9041829a79027dec81adc3 P ece5590526f94058b07afe5bb610ccfa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ece5590526f94058b07afe5bb610ccfa, State: Initialized, Role: FOLLOWER
I20260929 17:53:37.141185 8451 consensus_queue.cc:260] T 9c049e4dfa9041829a79027dec81adc3 P ece5590526f94058b07afe5bb610ccfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ece5590526f94058b07afe5bb610ccfa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40515 } } peers { permanent_uuid: "4e821d34d1844842be7365e97204614e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43829 } } peers { permanent_uuid: "c8be608a8c8b43e6a7063022a8a7da36" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32913 } }
I20260929 17:53:37.141436 8451 ts_tablet_manager.cc:1428] T 9c049e4dfa9041829a79027dec81adc3 P ece5590526f94058b07afe5bb610ccfa: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 17:53:37.141728 8450 tablet_bootstrap.cc:492] T 9c049e4dfa9041829a79027dec81adc3 P c8be608a8c8b43e6a7063022a8a7da36: Bootstrap starting.
I20260929 17:53:37.142266 8449 tablet_bootstrap.cc:492] T 9c049e4dfa9041829a79027dec81adc3 P 4e821d34d1844842be7365e97204614e: No bootstrap required, opened a new log
I20260929 17:53:37.142341 8449 ts_tablet_manager.cc:1397] T 9c049e4dfa9041829a79027dec81adc3 P 4e821d34d1844842be7365e97204614e: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260929 17:53:37.142477 8450 tablet_bootstrap.cc:654] T 9c049e4dfa9041829a79027dec81adc3 P c8be608a8c8b43e6a7063022a8a7da36: Neither blocks nor log segments found. Creating new log.
I20260929 17:53:37.142686 8449 raft_consensus.cc:348] T 9c049e4dfa9041829a79027dec81adc3 P 4e821d34d1844842be7365e97204614e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ece5590526f94058b07afe5bb610ccfa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40515 } } peers { permanent_uuid: "4e821d34d1844842be7365e97204614e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43829 } } peers { permanent_uuid: "c8be608a8c8b43e6a7063022a8a7da36" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32913 } }
I20260929 17:53:37.142771 8449 raft_consensus.cc:374] T 9c049e4dfa9041829a79027dec81adc3 P 4e821d34d1844842be7365e97204614e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:53:37.142807 8449 raft_consensus.cc:729] T 9c049e4dfa9041829a79027dec81adc3 P 4e821d34d1844842be7365e97204614e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e821d34d1844842be7365e97204614e, State: Initialized, Role: FOLLOWER
I20260929 17:53:37.142922 8449 consensus_queue.cc:260] T 9c049e4dfa9041829a79027dec81adc3 P 4e821d34d1844842be7365e97204614e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ece5590526f94058b07afe5bb610ccfa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40515 } } peers { permanent_uuid: "4e821d34d1844842be7365e97204614e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43829 } } peers { permanent_uuid: "c8be608a8c8b43e6a7063022a8a7da36" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32913 } }
I20260929 17:53:37.143146 8449 ts_tablet_manager.cc:1428] T 9c049e4dfa9041829a79027dec81adc3 P 4e821d34d1844842be7365e97204614e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 17:53:37.143883 8450 tablet_bootstrap.cc:492] T 9c049e4dfa9041829a79027dec81adc3 P c8be608a8c8b43e6a7063022a8a7da36: No bootstrap required, opened a new log
I20260929 17:53:37.143959 8450 ts_tablet_manager.cc:1397] T 9c049e4dfa9041829a79027dec81adc3 P c8be608a8c8b43e6a7063022a8a7da36: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 17:53:37.144328 8450 raft_consensus.cc:348] T 9c049e4dfa9041829a79027dec81adc3 P c8be608a8c8b43e6a7063022a8a7da36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ece5590526f94058b07afe5bb610ccfa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40515 } } peers { permanent_uuid: "4e821d34d1844842be7365e97204614e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43829 } } peers { permanent_uuid: "c8be608a8c8b43e6a7063022a8a7da36" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32913 } }
I20260929 17:53:37.144435 8450 raft_consensus.cc:374] T 9c049e4dfa9041829a79027dec81adc3 P c8be608a8c8b43e6a7063022a8a7da36 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:53:37.144471 8450 raft_consensus.cc:729] T 9c049e4dfa9041829a79027dec81adc3 P c8be608a8c8b43e6a7063022a8a7da36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8be608a8c8b43e6a7063022a8a7da36, State: Initialized, Role: FOLLOWER
I20260929 17:53:37.144589 8450 consensus_queue.cc:260] T 9c049e4dfa9041829a79027dec81adc3 P c8be608a8c8b43e6a7063022a8a7da36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ece5590526f94058b07afe5bb610ccfa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40515 } } peers { permanent_uuid: "4e821d34d1844842be7365e97204614e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43829 } } peers { permanent_uuid: "c8be608a8c8b43e6a7063022a8a7da36" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32913 } }
I20260929 17:53:37.144852 8450 ts_tablet_manager.cc:1428] T 9c049e4dfa9041829a79027dec81adc3 P c8be608a8c8b43e6a7063022a8a7da36: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 17:53:37.202281 8528 raft_consensus.cc:482] T 9c049e4dfa9041829a79027dec81adc3 P ece5590526f94058b07afe5bb610ccfa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 17:53:37.202430 8528 raft_consensus.cc:504] T 9c049e4dfa9041829a79027dec81adc3 P ece5590526f94058b07afe5bb610ccfa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ece5590526f94058b07afe5bb610ccfa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40515 } } peers { permanent_uuid: "4e821d34d1844842be7365e97204614e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43829 } } peers { permanent_uuid: "c8be608a8c8b43e6a7063022a8a7da36" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32913 } }
I20260929 17:53:37.202822 8528 leader_election.cc:288] T 9c049e4dfa9041829a79027dec81adc3 P ece5590526f94058b07afe5bb610ccfa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4e821d34d1844842be7365e97204614e (127.0.98.2:43829), c8be608a8c8b43e6a7063022a8a7da36 (127.0.98.3:32913)
I20260929 17:53:37.203258 8375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c049e4dfa9041829a79027dec81adc3" candidate_uuid: "ece5590526f94058b07afe5bb610ccfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8be608a8c8b43e6a7063022a8a7da36" is_pre_election: true
I20260929 17:53:37.203258 8235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c049e4dfa9041829a79027dec81adc3" candidate_uuid: "ece5590526f94058b07afe5bb610ccfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e821d34d1844842be7365e97204614e" is_pre_election: true
I20260929 17:53:37.203394 8235 raft_consensus.cc:2454] T 9c049e4dfa9041829a79027dec81adc3 P 4e821d34d1844842be7365e97204614e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ece5590526f94058b07afe5bb610ccfa in term 0.
I20260929 17:53:37.203402 8375 raft_consensus.cc:2454] T 9c049e4dfa9041829a79027dec81adc3 P c8be608a8c8b43e6a7063022a8a7da36 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ece5590526f94058b07afe5bb610ccfa in term 0.
I20260929 17:53:37.203658 8029 leader_election.cc:302] T 9c049e4dfa9041829a79027dec81adc3 P ece5590526f94058b07afe5bb610ccfa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e821d34d1844842be7365e97204614e, ece5590526f94058b07afe5bb610ccfa; no voters:
I20260929 17:53:37.203925 8528 raft_consensus.cc:2781] T 9c049e4dfa9041829a79027dec81adc3 P ece5590526f94058b07afe5bb610ccfa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 17:53:37.203989 8528 raft_consensus.cc:482] T 9c049e4dfa9041829a79027dec81adc3 P ece5590526f94058b07afe5bb610ccfa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 17:53:37.204031 8528 raft_consensus.cc:3037] T 9c049e4dfa9041829a79027dec81adc3 P ece5590526f94058b07afe5bb610ccfa [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:53:37.205325 8528 raft_consensus.cc:504] T 9c049e4dfa9041829a79027dec81adc3 P ece5590526f94058b07afe5bb610ccfa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ece5590526f94058b07afe5bb610ccfa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40515 } } peers { permanent_uuid: "4e821d34d1844842be7365e97204614e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43829 } } peers { permanent_uuid: "c8be608a8c8b43e6a7063022a8a7da36" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32913 } }
I20260929 17:53:37.205664 8528 leader_election.cc:288] T 9c049e4dfa9041829a79027dec81adc3 P ece5590526f94058b07afe5bb610ccfa [CANDIDATE]: Term 1 election: Requested vote from peers 4e821d34d1844842be7365e97204614e (127.0.98.2:43829), c8be608a8c8b43e6a7063022a8a7da36 (127.0.98.3:32913)
I20260929 17:53:37.206084 8375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c049e4dfa9041829a79027dec81adc3" candidate_uuid: "ece5590526f94058b07afe5bb610ccfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8be608a8c8b43e6a7063022a8a7da36"
I20260929 17:53:37.206205 8375 raft_consensus.cc:3037] T 9c049e4dfa9041829a79027dec81adc3 P c8be608a8c8b43e6a7063022a8a7da36 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:53:37.207331 8375 raft_consensus.cc:2454] T 9c049e4dfa9041829a79027dec81adc3 P c8be608a8c8b43e6a7063022a8a7da36 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ece5590526f94058b07afe5bb610ccfa in term 1.
I20260929 17:53:37.207613 8029 leader_election.cc:302] T 9c049e4dfa9041829a79027dec81adc3 P ece5590526f94058b07afe5bb610ccfa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c8be608a8c8b43e6a7063022a8a7da36, ece5590526f94058b07afe5bb610ccfa; no voters:
I20260929 17:53:37.207595 8235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c049e4dfa9041829a79027dec81adc3" candidate_uuid: "ece5590526f94058b07afe5bb610ccfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e821d34d1844842be7365e97204614e"
I20260929 17:53:37.207707 8235 raft_consensus.cc:3037] T 9c049e4dfa9041829a79027dec81adc3 P 4e821d34d1844842be7365e97204614e [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:53:37.207772 8528 raft_consensus.cc:2781] T 9c049e4dfa9041829a79027dec81adc3 P ece5590526f94058b07afe5bb610ccfa [term 1 FOLLOWER]: Leader election won for term 1
I20260929 17:53:37.207852 8528 raft_consensus.cc:686] T 9c049e4dfa9041829a79027dec81adc3 P ece5590526f94058b07afe5bb610ccfa [term 1 LEADER]: Becoming Leader. State: Replica: ece5590526f94058b07afe5bb610ccfa, State: Running, Role: LEADER
I20260929 17:53:37.207985 8528 consensus_queue.cc:237] T 9c049e4dfa9041829a79027dec81adc3 P ece5590526f94058b07afe5bb610ccfa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ece5590526f94058b07afe5bb610ccfa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40515 } } peers { permanent_uuid: "4e821d34d1844842be7365e97204614e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43829 } } peers { permanent_uuid: "c8be608a8c8b43e6a7063022a8a7da36" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32913 } }
I20260929 17:53:37.208788 8235 raft_consensus.cc:2454] T 9c049e4dfa9041829a79027dec81adc3 P 4e821d34d1844842be7365e97204614e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ece5590526f94058b07afe5bb610ccfa in term 1.
I20260929 17:53:37.209757 7937 catalog_manager.cc:5495] T 9c049e4dfa9041829a79027dec81adc3 P ece5590526f94058b07afe5bb610ccfa reported cstate change: term changed from 0 to 1, leader changed from <none> to ece5590526f94058b07afe5bb610ccfa (127.0.98.1). New cstate: current_term: 1 leader_uuid: "ece5590526f94058b07afe5bb610ccfa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ece5590526f94058b07afe5bb610ccfa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40515 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4e821d34d1844842be7365e97204614e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43829 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8be608a8c8b43e6a7063022a8a7da36" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32913 } health_report { overall_health: UNKNOWN } } }
I20260929 17:53:37.235736 8448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:37.232053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36551 (local address 127.0.98.2:43829)
0929 17:53:37.232180 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:37.232185 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:37.232452 (+ 267us) server_negotiation.cc:407] Connection header received
0929 17:53:37.232586 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:37.232588 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:37.232632 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:37.232696 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 17:53:37.232982 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:37.233528 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:37.234762 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:37.234967 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:37.235033 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:37.235364 (+ 331us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 17:53:37.235429 (+ 65us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 17:53:37.235470 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0929 17:53:37.235582 (+ 112us) server_negotiation.cc:299] Negotiation successful
0929 17:53:37.235625 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260929 17:53:37.235867 8536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:37.231946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43829 (local address 127.0.98.1:36551)
0929 17:53:37.232357 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:37.232372 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 17:53:37.232493 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:53:37.232725 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:53:37.232728 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:53:37.232737 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 17:53:37.232888 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:37.232895 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:37.233636 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:37.233640 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:37.234654 (+ 1014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:37.234666 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:37.235080 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:37.235084 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:37.235185 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:37.235236 (+ 51us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 17:53:37.235497 (+ 261us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 17:53:37.235504 (+ 7us) client_negotiation.cc:769] Sending connection context
0929 17:53:37.235578 (+ 74us) client_negotiation.cc:240] Negotiation successful
0929 17:53:37.235663 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":109,"threads_started":1}
I20260929 17:53:37.241842 8235 raft_consensus.cc:1261] T 9c049e4dfa9041829a79027dec81adc3 P 4e821d34d1844842be7365e97204614e [term 1 FOLLOWER]: Refusing update from remote peer ece5590526f94058b07afe5bb610ccfa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 17:53:37.241842 8375 raft_consensus.cc:1261] T 9c049e4dfa9041829a79027dec81adc3 P c8be608a8c8b43e6a7063022a8a7da36 [term 1 FOLLOWER]: Refusing update from remote peer ece5590526f94058b07afe5bb610ccfa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 17:53:37.242254 8457 consensus_queue.cc:1035] T 9c049e4dfa9041829a79027dec81adc3 P ece5590526f94058b07afe5bb610ccfa [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e821d34d1844842be7365e97204614e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 17:53:37.242460 8528 consensus_queue.cc:1035] T 9c049e4dfa9041829a79027dec81adc3 P ece5590526f94058b07afe5bb610ccfa [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8be608a8c8b43e6a7063022a8a7da36" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 17:53:37.259073 8447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:37.255446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42971 (local address 127.0.98.1:40515)
0929 17:53:37.255557 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:37.255560 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:37.255572 (+ 12us) server_negotiation.cc:407] Connection header received
0929 17:53:37.255613 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:37.255616 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:37.255668 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:37.255727 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 17:53:37.256044 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:37.256799 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:37.257869 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:37.258220 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:37.258332 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:37.258646 (+ 314us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 17:53:37.258729 (+ 83us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 17:53:37.258794 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0929 17:53:37.258914 (+ 120us) server_negotiation.cc:299] Negotiation successful
0929 17:53:37.258956 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260929 17:53:37.259146 8504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:37.255212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40515 (local address 127.0.98.2:42971)
0929 17:53:37.255439 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:37.255455 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 17:53:37.255533 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:53:37.255839 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:53:37.255842 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:53:37.255850 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 17:53:37.255945 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:37.255951 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:37.256914 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:37.256916 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:37.257735 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:37.257745 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:37.258413 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:37.258417 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:37.258501 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:37.258532 (+ 31us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 17:53:37.258820 (+ 288us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 17:53:37.258828 (+ 8us) client_negotiation.cc:769] Sending connection context
0929 17:53:37.258907 (+ 79us) client_negotiation.cc:240] Negotiation successful
0929 17:53:37.258997 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139}
I20260929 17:53:37.276556 7938 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51962:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 17:53:37.276721 7938 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51962:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 17:53:37.278697 7938 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1fb78915f2ed49e5b241da7350b145ae: Sending DeleteTablet for 3 replicas of tablet 9c049e4dfa9041829a79027dec81adc3
I20260929 17:53:37.279170 8075 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c049e4dfa9041829a79027dec81adc3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 17:53:37 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:32840
I20260929 17:53:37.279210 8215 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c049e4dfa9041829a79027dec81adc3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 17:53:37 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45988
I20260929 17:53:37.279348 8472 tablet_replica.cc:330] T 9c049e4dfa9041829a79027dec81adc3 P ece5590526f94058b07afe5bb610ccfa: stopping tablet replica
I20260929 17:53:37.279361 8355 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c049e4dfa9041829a79027dec81adc3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 17:53:37 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43864
I20260929 17:53:37.279444 8472 raft_consensus.cc:2229] T 9c049e4dfa9041829a79027dec81adc3 P ece5590526f94058b07afe5bb610ccfa [term 1 LEADER]: Raft consensus shutting down.
I20260929 17:53:37.279619 8472 raft_consensus.cc:2258] T 9c049e4dfa9041829a79027dec81adc3 P ece5590526f94058b07afe5bb610ccfa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 17:53:37.279803 8473 tablet_replica.cc:330] T 9c049e4dfa9041829a79027dec81adc3 P 4e821d34d1844842be7365e97204614e: stopping tablet replica
I20260929 17:53:37.279912 8473 raft_consensus.cc:2229] T 9c049e4dfa9041829a79027dec81adc3 P 4e821d34d1844842be7365e97204614e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 17:53:37.280014 8473 raft_consensus.cc:2258] T 9c049e4dfa9041829a79027dec81adc3 P 4e821d34d1844842be7365e97204614e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 17:53:37.280167 8474 tablet_replica.cc:330] T 9c049e4dfa9041829a79027dec81adc3 P c8be608a8c8b43e6a7063022a8a7da36: stopping tablet replica
I20260929 17:53:37.280275 8474 raft_consensus.cc:2229] T 9c049e4dfa9041829a79027dec81adc3 P c8be608a8c8b43e6a7063022a8a7da36 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 17:53:37.280375 8474 raft_consensus.cc:2258] T 9c049e4dfa9041829a79027dec81adc3 P c8be608a8c8b43e6a7063022a8a7da36 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 17:53:37.280614 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskup3Soe/build/debug/bin/kudu with pid 8010
I20260929 17:53:37.281585 8473 ts_tablet_manager.cc:1905] T 9c049e4dfa9041829a79027dec81adc3 P 4e821d34d1844842be7365e97204614e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 17:53:37.283641 8473 ts_tablet_manager.cc:1918] T 9c049e4dfa9041829a79027dec81adc3 P 4e821d34d1844842be7365e97204614e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 17:53:37.283710 8473 log.cc:1196] T 9c049e4dfa9041829a79027dec81adc3 P 4e821d34d1844842be7365e97204614e: Deleting WAL directory at /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704375975685-392-0/minicluster-data/ts-1/wal/wals/9c049e4dfa9041829a79027dec81adc3
I20260929 17:53:37.283917 8473 ts_tablet_manager.cc:1939] T 9c049e4dfa9041829a79027dec81adc3 P 4e821d34d1844842be7365e97204614e: Deleting consensus metadata
I20260929 17:53:37.284494 7925 catalog_manager.cc:4841] TS 4e821d34d1844842be7365e97204614e (127.0.98.2:43829): tablet 9c049e4dfa9041829a79027dec81adc3 (table test-table [id=538ec8f33f874839a31ff2415e711eeb]) successfully deleted
W20260929 17:53:37.287535 7922 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:40515 (error 108)
I20260929 17:53:37.287652 8474 ts_tablet_manager.cc:1905] T 9c049e4dfa9041829a79027dec81adc3 P c8be608a8c8b43e6a7063022a8a7da36: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 17:53:37.287787 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskup3Soe/build/debug/bin/kudu with pid 8150
I20260929 17:53:37.289883 8474 ts_tablet_manager.cc:1918] T 9c049e4dfa9041829a79027dec81adc3 P c8be608a8c8b43e6a7063022a8a7da36: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 17:53:37.289963 8474 log.cc:1196] T 9c049e4dfa9041829a79027dec81adc3 P c8be608a8c8b43e6a7063022a8a7da36: Deleting WAL directory at /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790704375975685-392-0/minicluster-data/ts-2/wal/wals/9c049e4dfa9041829a79027dec81adc3
I20260929 17:53:37.290072 8540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:37.289667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40515 (local address 127.0.0.1:32862)
0929 17:53:37.289893 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:37.289974 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40515: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":58,"threads_started":1}
I20260929 17:53:37.290194 8474 ts_tablet_manager.cc:1939] T 9c049e4dfa9041829a79027dec81adc3 P c8be608a8c8b43e6a7063022a8a7da36: Deleting consensus metadata
W20260929 17:53:37.290203 7922 catalog_manager.cc:4584] TS ece5590526f94058b07afe5bb610ccfa (127.0.98.1:40515): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9c049e4dfa9041829a79027dec81adc3: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40515: connect: Connection refused (error 111)
I20260929 17:53:37.290841 7925 catalog_manager.cc:4841] TS c8be608a8c8b43e6a7063022a8a7da36 (127.0.98.3:32913): tablet 9c049e4dfa9041829a79027dec81adc3 (table test-table [id=538ec8f33f874839a31ff2415e711eeb]) successfully deleted
I20260929 17:53:37.294428 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskup3Soe/build/debug/bin/kudu with pid 8290
I20260929 17:53:37.299973 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskup3Soe/build/debug/bin/kudu with pid 7906
2026-09-29T17:53:37Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (2876 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790704375975685-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 17:53:37 dist-test-slave-53nh krb5kdc[8543](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 17:53:37 dist-test-slave-53nh krb5kdc[8543](info): set up 2 sockets
Sep 29 17:53:37 dist-test-slave-53nh krb5kdc[8543](info): commencing operation
krb5kdc: starting...
W20260929 17:53:38.249580 392 mini_kdc.cc:121] Time spent starting KDC: real 0.928s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790704375975685-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790704375975685-392-0/krb5kdc/test-user.keytab.
Sep 29 17:53:38 dist-test-slave-53nh krb5kdc[8543](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704418, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T17:53:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T17:53:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790704375975685-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790704375975685-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 17:53:38.361223 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskup3Soe/build/debug/bin/kudu
/tmp/dist-test-taskup3Soe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790704375975685-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790704375975685-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790704375975685-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790704375975685-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34683
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790704375975685-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38019
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34683
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790704375975685-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790704375975685-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790704375975685-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:53:38.469184 8556 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:53:38.469470 8556 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:53:38.469524 8556 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:53:38.473526 8556 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 17:53:38.473590 8556 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:53:38.473616 8556 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:53:38.473636 8556 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 17:53:38.473656 8556 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 17:53:38.479020 8556 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38019
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790704375975685-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790704375975685-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34683
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790704375975685-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34683
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790704375975685-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8556
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790704375975685-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:53:38.480111 8556 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:53:38.481143 8556 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:53:38.489534 8564 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:53:38.489531 8562 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:53:38.489562 8561 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:53:38.489833 8556 server_base.cc:1034] running on GCE node
I20260929 17:53:38.490567 8556 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:53:38.491729 8556 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:53:38.492897 8556 hybrid_clock.cc:648] HybridClock initialized: now 1790704418492877 us; error 34 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-29T17:53:38Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1199 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704375975685-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 17:53:38 dist-test-slave-53nh krb5kdc[8569](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 17:53:38 dist-test-slave-53nh krb5kdc[8569](info): set up 2 sockets
Sep 29 17:53:38 dist-test-slave-53nh krb5kdc[8569](info): commencing operation
krb5kdc: starting...
W20260929 17:53:39.453451 392 mini_kdc.cc:121] Time spent starting KDC: real 0.933s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704375975685-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704375975685-392-0/krb5kdc/test-user.keytab.
Sep 29 17:53:39 dist-test-slave-53nh krb5kdc[8569](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704419, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T17:53:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T17:53:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704375975685-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704375975685-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 17:53:39.566048 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskup3Soe/build/debug/bin/kudu
/tmp/dist-test-taskup3Soe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704375975685-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704375975685-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704375975685-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704375975685-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38589
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704375975685-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38589
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704375975685-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704375975685-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704375975685-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:53:39.675067 8582 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:53:39.675400 8582 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:53:39.675442 8582 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:53:39.679658 8582 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 17:53:39.679729 8582 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:53:39.679749 8582 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:53:39.679764 8582 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 17:53:39.679778 8582 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 17:53:39.685226 8582 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704375975685-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704375975685-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38589
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704375975685-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38589
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704375975685-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8582
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704375975685-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:53:39.686342 8582 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:53:39.687386 8582 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:53:39.695458 8590 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:53:39.695468 8587 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:53:39.695489 8588 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:53:39.695789 8582 server_base.cc:1034] running on GCE node
I20260929 17:53:39.696324 8582 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:53:39.697450 8582 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:53:39.698612 8582 hybrid_clock.cc:648] HybridClock initialized: now 1790704419698597 us; error 29 us; skew 500 ppm
Sep 29 17:53:39 dist-test-slave-53nh krb5kdc[8569](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704419, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:53:39.707654 8582 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 17:53:39.708660 8582 webserver.cc:460] Webserver started at http://127.0.98.62:41871/ using document root <none> and password file <none>
I20260929 17:53:39.709311 8582 fs_manager.cc:359] Metadata directory not provided
I20260929 17:53:39.709385 8582 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:53:39.709586 8582 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:53:39.711679 8582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704375975685-392-0/minicluster-data/master-0/data/instance:
uuid: "672d23dbfaea49e78e0cc754dbc16953"
format_stamp: "Formatted at 2026-09-29 17:53:39 on dist-test-slave-53nh"
server_key: "1b1552f22a33d2f1459c0298ef2dbe82"
server_key_iv: "a080c65d54bd0cd54361a4cc8e9f1842"
server_key_version: "encryptionkey@0"
I20260929 17:53:39.712201 8582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704375975685-392-0/minicluster-data/master-0/wal/instance:
uuid: "672d23dbfaea49e78e0cc754dbc16953"
format_stamp: "Formatted at 2026-09-29 17:53:39 on dist-test-slave-53nh"
server_key: "1b1552f22a33d2f1459c0298ef2dbe82"
server_key_iv: "a080c65d54bd0cd54361a4cc8e9f1842"
server_key_version: "encryptionkey@0"
I20260929 17:53:39.715906 8582 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260929 17:53:39.718497 8597 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:53:39.719815 8582 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 17:53:39.719936 8582 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704375975685-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704375975685-392-0/minicluster-data/master-0/wal
uuid: "672d23dbfaea49e78e0cc754dbc16953"
format_stamp: "Formatted at 2026-09-29 17:53:39 on dist-test-slave-53nh"
server_key: "1b1552f22a33d2f1459c0298ef2dbe82"
server_key_iv: "a080c65d54bd0cd54361a4cc8e9f1842"
server_key_version: "encryptionkey@0"
I20260929 17:53:39.720041 8582 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704375975685-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704375975685-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704375975685-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:53:39.749091 8582 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:53:39.752067 8582 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:53:39.752266 8582 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:53:39.759268 8582 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38589
I20260929 17:53:39.759311 8649 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38589 every 8 connection(s)
I20260929 17:53:39.760311 8582 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704375975685-392-0/minicluster-data/master-0/data/info.pb
I20260929 17:53:39.762549 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskup3Soe/build/debug/bin/kudu as pid 8582
I20260929 17:53:39.762667 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704375975685-392-0/minicluster-data/master-0/wal/instance
I20260929 17:53:39.762930 392 external_mini_cluster.cc:1428] Setting key 313f78d80019f8db6fb628b2c50794a8
I20260929 17:53:39.763537 8650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:53:39.770488 8650 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 672d23dbfaea49e78e0cc754dbc16953: Bootstrap starting.
Sep 29 17:53:39 dist-test-slave-53nh krb5kdc[8569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704419, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 17:53:39.773228 8650 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 672d23dbfaea49e78e0cc754dbc16953: Neither blocks nor log segments found. Creating new log.
I20260929 17:53:39.774125 8650 log.cc:824] T 00000000000000000000000000000000 P 672d23dbfaea49e78e0cc754dbc16953: Log is configured to *not* fsync() on all Append() calls
I20260929 17:53:39.776378 8650 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 672d23dbfaea49e78e0cc754dbc16953: No bootstrap required, opened a new log
I20260929 17:53:39.777879 8653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:39.764427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42342 (local address 127.0.98.62:38589)
0929 17:53:39.764847 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:39.764863 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:39.764898 (+ 35us) server_negotiation.cc:407] Connection header received
0929 17:53:39.765589 (+ 691us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:39.765607 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:39.765939 (+ 332us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:39.766218 (+ 279us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:53:39.767199 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:39.768208 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:39.769485 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:39.769968 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:39.770084 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:39.772926 (+ 2842us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:53:39.772930 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:53:39.772945 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:53:39.772973 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:53:39.774829 (+ 1856us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:53:39.775242 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:53:39.775248 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:53:39.775254 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:53:39.775398 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:53:39.775635 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:53:39.775638 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:53:39.775640 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:53:39.776158 (+ 518us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:53:39.776356 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0929 17:53:39.776570 (+ 214us) server_negotiation.cc:299] Negotiation successful
0929 17:53:39.776885 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282,"thread_start_us":134,"threads_started":1}
I20260929 17:53:39.780720 8650 raft_consensus.cc:348] T 00000000000000000000000000000000 P 672d23dbfaea49e78e0cc754dbc16953 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "672d23dbfaea49e78e0cc754dbc16953" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38589 } }
I20260929 17:53:39.780947 8650 raft_consensus.cc:374] T 00000000000000000000000000000000 P 672d23dbfaea49e78e0cc754dbc16953 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:53:39.780992 8650 raft_consensus.cc:729] T 00000000000000000000000000000000 P 672d23dbfaea49e78e0cc754dbc16953 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 672d23dbfaea49e78e0cc754dbc16953, State: Initialized, Role: FOLLOWER
I20260929 17:53:39.781436 8650 consensus_queue.cc:260] T 00000000000000000000000000000000 P 672d23dbfaea49e78e0cc754dbc16953 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "672d23dbfaea49e78e0cc754dbc16953" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38589 } }
I20260929 17:53:39.781576 8650 raft_consensus.cc:388] T 00000000000000000000000000000000 P 672d23dbfaea49e78e0cc754dbc16953 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 17:53:39.781628 8650 raft_consensus.cc:482] T 00000000000000000000000000000000 P 672d23dbfaea49e78e0cc754dbc16953 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 17:53:39.781704 8650 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 672d23dbfaea49e78e0cc754dbc16953 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:53:39.782647 8650 raft_consensus.cc:504] T 00000000000000000000000000000000 P 672d23dbfaea49e78e0cc754dbc16953 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "672d23dbfaea49e78e0cc754dbc16953" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38589 } }
I20260929 17:53:39.782961 8650 leader_election.cc:302] T 00000000000000000000000000000000 P 672d23dbfaea49e78e0cc754dbc16953 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 672d23dbfaea49e78e0cc754dbc16953; no voters:
I20260929 17:53:39.783245 8650 leader_election.cc:288] T 00000000000000000000000000000000 P 672d23dbfaea49e78e0cc754dbc16953 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 17:53:39.783435 8655 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 672d23dbfaea49e78e0cc754dbc16953 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 17:53:39.783640 8655 raft_consensus.cc:686] T 00000000000000000000000000000000 P 672d23dbfaea49e78e0cc754dbc16953 [term 1 LEADER]: Becoming Leader. State: Replica: 672d23dbfaea49e78e0cc754dbc16953, State: Running, Role: LEADER
I20260929 17:53:39.783937 8655 consensus_queue.cc:237] T 00000000000000000000000000000000 P 672d23dbfaea49e78e0cc754dbc16953 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "672d23dbfaea49e78e0cc754dbc16953" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38589 } }
I20260929 17:53:39.784500 8650 sys_catalog.cc:564] T 00000000000000000000000000000000 P 672d23dbfaea49e78e0cc754dbc16953 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 17:53:39.785658 8656 sys_catalog.cc:455] T 00000000000000000000000000000000 P 672d23dbfaea49e78e0cc754dbc16953 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "672d23dbfaea49e78e0cc754dbc16953" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "672d23dbfaea49e78e0cc754dbc16953" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38589 } } }
I20260929 17:53:39.785688 8657 sys_catalog.cc:455] T 00000000000000000000000000000000 P 672d23dbfaea49e78e0cc754dbc16953 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 672d23dbfaea49e78e0cc754dbc16953. Latest consensus state: current_term: 1 leader_uuid: "672d23dbfaea49e78e0cc754dbc16953" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "672d23dbfaea49e78e0cc754dbc16953" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38589 } } }
I20260929 17:53:39.785820 8656 sys_catalog.cc:458] T 00000000000000000000000000000000 P 672d23dbfaea49e78e0cc754dbc16953 [sys.catalog]: This master's current role is: LEADER
I20260929 17:53:39.785820 8657 sys_catalog.cc:458] T 00000000000000000000000000000000 P 672d23dbfaea49e78e0cc754dbc16953 [sys.catalog]: This master's current role is: LEADER
I20260929 17:53:39.786218 8664 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 17:53:39.789180 8664 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 17:53:39.794445 8664 catalog_manager.cc:1347] Generated new cluster ID: a5a9de8f68d148e989e5be49b9242f83
I20260929 17:53:39.794513 8664 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 17:53:39.826159 8664 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 17:53:39.827534 8664 catalog_manager.cc:1504] Loading token signing keys...
I20260929 17:53:39.832528 8664 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 672d23dbfaea49e78e0cc754dbc16953: Generated new TSK 0
I20260929 17:53:39.833439 8664 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704375975685-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704375975685-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260929 17:53:39.872277 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskup3Soe/build/debug/bin/kudu
/tmp/dist-test-taskup3Soe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704375975685-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704375975685-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704375975685-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704375975685-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38589
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704375975685-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704375975685-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704375975685-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704375975685-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:53:39.979713 8676 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:53:39.979975 8676 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:53:39.980017 8676 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:53:39.983966 8676 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:53:39.984021 8676 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:53:39.984076 8676 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 17:53:39.989390 8676 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704375975685-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704375975685-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704375975685-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704375975685-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38589
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8676
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704375975685-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:53:39.990360 8676 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:53:39.991307 8676 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:53:40.000491 8682 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:53:40.000613 8676 server_base.cc:1034] running on GCE node
W20260929 17:53:40.000491 8681 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:53:40.000511 8684 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:53:40.001253 8676 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:53:40.001933 8676 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:53:40.003110 8676 hybrid_clock.cc:648] HybridClock initialized: now 1790704420003071 us; error 50 us; skew 500 ppm
Sep 29 17:53:40 dist-test-slave-53nh krb5kdc[8569](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704420, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:53:40.005858 8676 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260929 17:53:40.006753 8676 webserver.cc:460] Webserver started at http://127.0.98.1:35681/ using document root <none> and password file <none>
I20260929 17:53:40.007282 8676 fs_manager.cc:359] Metadata directory not provided
I20260929 17:53:40.007359 8676 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:53:40.007548 8676 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:53:40.009584 8676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704375975685-392-0/minicluster-data/ts-0/data/instance:
uuid: "3c5cb2ae7921438788cb6ee7df2aa560"
format_stamp: "Formatted at 2026-09-29 17:53:40 on dist-test-slave-53nh"
server_key: "46d9ba2c24a2a4aa30943bffc149e6e3"
server_key_iv: "0477e495340560c7092e6e0545534bf3"
server_key_version: "encryptionkey@0"
I20260929 17:53:40.010080 8676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704375975685-392-0/minicluster-data/ts-0/wal/instance:
uuid: "3c5cb2ae7921438788cb6ee7df2aa560"
format_stamp: "Formatted at 2026-09-29 17:53:40 on dist-test-slave-53nh"
server_key: "46d9ba2c24a2a4aa30943bffc149e6e3"
server_key_iv: "0477e495340560c7092e6e0545534bf3"
server_key_version: "encryptionkey@0"
I20260929 17:53:40.013622 8676 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 17:53:40.016002 8691 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:53:40.017267 8676 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 17:53:40.017360 8676 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704375975685-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704375975685-392-0/minicluster-data/ts-0/wal
uuid: "3c5cb2ae7921438788cb6ee7df2aa560"
format_stamp: "Formatted at 2026-09-29 17:53:40 on dist-test-slave-53nh"
server_key: "46d9ba2c24a2a4aa30943bffc149e6e3"
server_key_iv: "0477e495340560c7092e6e0545534bf3"
server_key_version: "encryptionkey@0"
I20260929 17:53:40.017448 8676 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704375975685-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704375975685-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704375975685-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:53:40.026899 8676 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:53:40.029510 8676 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:53:40.029657 8676 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:53:40.030148 8676 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 17:53:40.030887 8676 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:53:40.030930 8676 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:53:40.030966 8676 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:53:40.030987 8676 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 17:53:40.039036 8676 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36233
I20260929 17:53:40.039063 8804 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36233 every 8 connection(s)
I20260929 17:53:40.039911 8676 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704375975685-392-0/minicluster-data/ts-0/data/info.pb
I20260929 17:53:40.048138 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskup3Soe/build/debug/bin/kudu as pid 8676
I20260929 17:53:40.048239 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704375975685-392-0/minicluster-data/ts-0/wal/instance
I20260929 17:53:40.048482 392 external_mini_cluster.cc:1428] Setting key 6cf390060e888e801abe11d5eb63ccc9
Sep 29 17:53:40 dist-test-slave-53nh krb5kdc[8569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704420, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 17:53:40.052834 8653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:40.041638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48231 (local address 127.0.98.62:38589)
0929 17:53:40.041771 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:40.041775 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:40.042600 (+ 825us) server_negotiation.cc:407] Connection header received
0929 17:53:40.043954 (+ 1354us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:40.043958 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:40.044003 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:40.044100 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:53:40.045271 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:40.045810 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:40.046873 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:40.047091 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:40.047160 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:40.049537 (+ 2377us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:53:40.049540 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:53:40.049542 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:53:40.049567 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:53:40.050938 (+ 1371us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:53:40.051419 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:53:40.051423 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:53:40.051425 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:53:40.051501 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:53:40.051800 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:53:40.051803 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:53:40.051805 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:53:40.051973 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:53:40.052059 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0929 17:53:40.052537 (+ 478us) server_negotiation.cc:299] Negotiation successful
0929 17:53:40.052671 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 17:53:40.053670 8807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:40.042010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38589 (local address 127.0.98.1:48231)
0929 17:53:40.042433 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:40.042476 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 17:53:40.043725 (+ 1249us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:53:40.044229 (+ 504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:53:40.044236 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:53:40.044605 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:53:40.045139 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:40.045155 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:40.045902 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:40.045904 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:40.046785 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:40.046793 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:40.047196 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:40.047198 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:40.047351 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:40.047896 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:53:40.047912 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:53:40.049383 (+ 1471us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:53:40.051043 (+ 1660us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:53:40.051048 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:53:40.051063 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:53:40.051327 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:53:40.051580 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:53:40.051583 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:53:40.051585 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:53:40.051717 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:53:40.052066 (+ 349us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:53:40.052073 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:53:40.052317 (+ 244us) client_negotiation.cc:769] Sending connection context
0929 17:53:40.052482 (+ 165us) client_negotiation.cc:240] Negotiation successful
0929 17:53:40.052801 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":102,"threads_started":1}
I20260929 17:53:40.055085 8805 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38589
I20260929 17:53:40.055325 8805 heartbeater.cc:464] Registering TS with master...
I20260929 17:53:40.055810 8805 heartbeater.cc:511] Master 127.0.98.62:38589 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260929 17:53:40.057462 8614 ts_manager.cc:194] Registered new tserver with Master: 3c5cb2ae7921438788cb6ee7df2aa560 (127.0.98.1:36233)
I20260929 17:53:40.058599 8614 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:48231
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704375975685-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704375975685-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260929 17:53:40.073904 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskup3Soe/build/debug/bin/kudu
/tmp/dist-test-taskup3Soe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704375975685-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704375975685-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704375975685-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704375975685-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38589
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704375975685-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704375975685-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704375975685-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704375975685-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:53:40.180280 8810 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:53:40.180553 8810 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:53:40.180609 8810 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:53:40.184669 8810 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:53:40.184727 8810 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:53:40.184808 8810 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 17:53:40.190055 8810 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704375975685-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704375975685-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704375975685-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704375975685-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38589
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8810
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704375975685-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:53:40.191041 8810 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:53:40.192026 8810 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:53:40.201112 8815 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:53:40.201097 8818 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:53:40.201097 8816 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:53:40.201346 8810 server_base.cc:1034] running on GCE node
I20260929 17:53:40.201844 8810 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:53:40.202450 8810 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:53:40.203604 8810 hybrid_clock.cc:648] HybridClock initialized: now 1790704420203583 us; error 39 us; skew 500 ppm
Sep 29 17:53:40 dist-test-slave-53nh krb5kdc[8569](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704420, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:53:40.206303 8810 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260929 17:53:40.207207 8810 webserver.cc:460] Webserver started at http://127.0.98.2:37541/ using document root <none> and password file <none>
I20260929 17:53:40.207726 8810 fs_manager.cc:359] Metadata directory not provided
I20260929 17:53:40.207795 8810 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:53:40.207991 8810 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:53:40.210052 8810 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704375975685-392-0/minicluster-data/ts-1/data/instance:
uuid: "a43890f8f55a40a2ac52fc685c42ed62"
format_stamp: "Formatted at 2026-09-29 17:53:40 on dist-test-slave-53nh"
server_key: "2d5ab5d65eba4670811c7777b9e739d7"
server_key_iv: "af34f9697014f9c407697b948a28d2a8"
server_key_version: "encryptionkey@0"
I20260929 17:53:40.210569 8810 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704375975685-392-0/minicluster-data/ts-1/wal/instance:
uuid: "a43890f8f55a40a2ac52fc685c42ed62"
format_stamp: "Formatted at 2026-09-29 17:53:40 on dist-test-slave-53nh"
server_key: "2d5ab5d65eba4670811c7777b9e739d7"
server_key_iv: "af34f9697014f9c407697b948a28d2a8"
server_key_version: "encryptionkey@0"
I20260929 17:53:40.214118 8810 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.004s
I20260929 17:53:40.216393 8825 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:53:40.217608 8810 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260929 17:53:40.217702 8810 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704375975685-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704375975685-392-0/minicluster-data/ts-1/wal
uuid: "a43890f8f55a40a2ac52fc685c42ed62"
format_stamp: "Formatted at 2026-09-29 17:53:40 on dist-test-slave-53nh"
server_key: "2d5ab5d65eba4670811c7777b9e739d7"
server_key_iv: "af34f9697014f9c407697b948a28d2a8"
server_key_version: "encryptionkey@0"
I20260929 17:53:40.217818 8810 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704375975685-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704375975685-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704375975685-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:53:40.238355 8810 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:53:40.241520 8810 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:53:40.241688 8810 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:53:40.242255 8810 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 17:53:40.243108 8810 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:53:40.243157 8810 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:53:40.243196 8810 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:53:40.243209 8810 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 17:53:40.251929 8810 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45123
I20260929 17:53:40.252055 8938 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45123 every 8 connection(s)
I20260929 17:53:40.252853 8810 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704375975685-392-0/minicluster-data/ts-1/data/info.pb
I20260929 17:53:40.259682 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskup3Soe/build/debug/bin/kudu as pid 8810
I20260929 17:53:40.259816 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704375975685-392-0/minicluster-data/ts-1/wal/instance
I20260929 17:53:40.260058 392 external_mini_cluster.cc:1428] Setting key 07709ffc74906c5aab365d5d93cd13fd
Sep 29 17:53:40 dist-test-slave-53nh krb5kdc[8569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704420, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 17:53:40.266512 8653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:40.254879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40583 (local address 127.0.98.62:38589)
0929 17:53:40.255006 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:40.255010 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:40.255932 (+ 922us) server_negotiation.cc:407] Connection header received
0929 17:53:40.257369 (+ 1437us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:40.257372 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:40.257428 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:40.257505 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:53:40.258715 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:40.259241 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:40.260427 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:40.260645 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:40.260742 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:40.263182 (+ 2440us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:53:40.263186 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:53:40.263187 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:53:40.263211 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:53:40.264461 (+ 1250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:53:40.264993 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:53:40.265020 (+ 27us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:53:40.265023 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:53:40.265100 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:53:40.265428 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:53:40.265430 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:53:40.265431 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:53:40.265630 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:53:40.265707 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0929 17:53:40.266291 (+ 584us) server_negotiation.cc:299] Negotiation successful
0929 17:53:40.266393 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 17:53:40.267685 8941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:40.255319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38589 (local address 127.0.98.2:40583)
0929 17:53:40.255778 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:40.255823 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 17:53:40.257152 (+ 1329us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:53:40.257691 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:53:40.257698 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:53:40.258051 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:53:40.258568 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:40.258581 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:40.259357 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:40.259360 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:40.260312 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:40.260322 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:40.260788 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:40.260791 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:40.260955 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:40.261513 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:53:40.261528 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:53:40.263035 (+ 1507us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:53:40.264571 (+ 1536us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:53:40.264577 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:53:40.264592 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:53:40.264913 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:53:40.265173 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:53:40.265176 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:53:40.265178 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:53:40.265348 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:53:40.265723 (+ 375us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:53:40.265730 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:53:40.266038 (+ 308us) client_negotiation.cc:769] Sending connection context
0929 17:53:40.266271 (+ 233us) client_negotiation.cc:240] Negotiation successful
0929 17:53:40.266612 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":105,"threads_started":1}
I20260929 17:53:40.268801 8939 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38589
I20260929 17:53:40.269079 8939 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260929 17:53:40.269646 8939 heartbeater.cc:511] Master 127.0.98.62:38589 requested a full tablet report, sending...
I20260929 17:53:40.270653 8614 ts_manager.cc:194] Registered new tserver with Master: a43890f8f55a40a2ac52fc685c42ed62 (127.0.98.2:45123)
I20260929 17:53:40.271203 8614 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:40583
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704375975685-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704375975685-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260929 17:53:40.285327 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskup3Soe/build/debug/bin/kudu
/tmp/dist-test-taskup3Soe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704375975685-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704375975685-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704375975685-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704375975685-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38589
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704375975685-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704375975685-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704375975685-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704375975685-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:53:40.390139 8944 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:53:40.390396 8944 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:53:40.390451 8944 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:53:40.394378 8944 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:53:40.394434 8944 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:53:40.394507 8944 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 17:53:40.399724 8944 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704375975685-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704375975685-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704375975685-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704375975685-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38589
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8944
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704375975685-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:53:40.400688 8944 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:53:40.401665 8944 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:53:40.410589 8952 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:53:40.410630 8950 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:53:40.410642 8949 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:53:40.410838 8944 server_base.cc:1034] running on GCE node
I20260929 17:53:40.411428 8944 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:53:40.412075 8944 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:53:40.413225 8944 hybrid_clock.cc:648] HybridClock initialized: now 1790704420413202 us; error 54 us; skew 500 ppm
Sep 29 17:53:40 dist-test-slave-53nh krb5kdc[8569](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704420, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:53:40.415925 8944 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260929 17:53:40.416836 8944 webserver.cc:460] Webserver started at http://127.0.98.3:33999/ using document root <none> and password file <none>
I20260929 17:53:40.417438 8944 fs_manager.cc:359] Metadata directory not provided
I20260929 17:53:40.417506 8944 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:53:40.417725 8944 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:53:40.419744 8944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704375975685-392-0/minicluster-data/ts-2/data/instance:
uuid: "d9adf54b683147b393258c2c82adc622"
format_stamp: "Formatted at 2026-09-29 17:53:40 on dist-test-slave-53nh"
server_key: "fc0dc88aef97cef4c0bfc0f9e6d7e224"
server_key_iv: "3647b780a2d3e5b845e08a47b145a5bd"
server_key_version: "encryptionkey@0"
I20260929 17:53:40.420297 8944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704375975685-392-0/minicluster-data/ts-2/wal/instance:
uuid: "d9adf54b683147b393258c2c82adc622"
format_stamp: "Formatted at 2026-09-29 17:53:40 on dist-test-slave-53nh"
server_key: "fc0dc88aef97cef4c0bfc0f9e6d7e224"
server_key_iv: "3647b780a2d3e5b845e08a47b145a5bd"
server_key_version: "encryptionkey@0"
I20260929 17:53:40.424089 8944 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 17:53:40.426628 8959 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:53:40.427871 8944 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 17:53:40.427980 8944 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704375975685-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704375975685-392-0/minicluster-data/ts-2/wal
uuid: "d9adf54b683147b393258c2c82adc622"
format_stamp: "Formatted at 2026-09-29 17:53:40 on dist-test-slave-53nh"
server_key: "fc0dc88aef97cef4c0bfc0f9e6d7e224"
server_key_iv: "3647b780a2d3e5b845e08a47b145a5bd"
server_key_version: "encryptionkey@0"
I20260929 17:53:40.428082 8944 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704375975685-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704375975685-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704375975685-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:53:40.438639 8944 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:53:40.441718 8944 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:53:40.441892 8944 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:53:40.442492 8944 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 17:53:40.443293 8944 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:53:40.443338 8944 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:53:40.443389 8944 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:53:40.443418 8944 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 17:53:40.451365 8944 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46099
I20260929 17:53:40.451390 9072 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46099 every 8 connection(s)
I20260929 17:53:40.452168 8944 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704375975685-392-0/minicluster-data/ts-2/data/info.pb
I20260929 17:53:40.460369 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskup3Soe/build/debug/bin/kudu as pid 8944
I20260929 17:53:40.460470 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704375975685-392-0/minicluster-data/ts-2/wal/instance
I20260929 17:53:40.460783 392 external_mini_cluster.cc:1428] Setting key d627e2a0c5bde4deea95ead3ccfdc80e
Sep 29 17:53:40 dist-test-slave-53nh krb5kdc[8569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704420, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 17:53:40.465150 8653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:40.453831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52631 (local address 127.0.98.62:38589)
0929 17:53:40.454005 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:40.454009 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:40.454781 (+ 772us) server_negotiation.cc:407] Connection header received
0929 17:53:40.456083 (+ 1302us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:40.456086 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:40.456140 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:40.456245 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:53:40.457417 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:40.457940 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:40.459025 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:40.459255 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:40.459338 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:40.461847 (+ 2509us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:53:40.461850 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:53:40.461852 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:53:40.461875 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:53:40.463139 (+ 1264us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:53:40.463596 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:53:40.463598 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:53:40.463600 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:53:40.463666 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:53:40.463980 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:53:40.463984 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:53:40.463985 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:53:40.464184 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:53:40.464392 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0929 17:53:40.464863 (+ 471us) server_negotiation.cc:299] Negotiation successful
0929 17:53:40.464973 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260929 17:53:40.465886 9075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:40.454179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38589 (local address 127.0.98.3:52631)
0929 17:53:40.454609 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:40.454650 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 17:53:40.455884 (+ 1234us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:53:40.456419 (+ 535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:53:40.456426 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:53:40.456768 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:53:40.457284 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:40.457297 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:40.458053 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:40.458055 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:40.458906 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:40.458916 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:40.459385 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:40.459389 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:40.459541 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:40.460066 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:53:40.460081 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:53:40.461701 (+ 1620us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:53:40.463249 (+ 1548us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:53:40.463266 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:53:40.463278 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:53:40.463524 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:53:40.463728 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:53:40.463730 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:53:40.463731 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:53:40.463871 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:53:40.464376 (+ 505us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:53:40.464381 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:53:40.464613 (+ 232us) client_negotiation.cc:769] Sending connection context
0929 17:53:40.464767 (+ 154us) client_negotiation.cc:240] Negotiation successful
0929 17:53:40.465069 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":98,"threads_started":1}
I20260929 17:53:40.466972 9073 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38589
I20260929 17:53:40.467176 9073 heartbeater.cc:464] Registering TS with master...
I20260929 17:53:40.467604 9073 heartbeater.cc:511] Master 127.0.98.62:38589 requested a full tablet report, sending...
I20260929 17:53:40.468562 8614 ts_manager.cc:194] Registered new tserver with Master: d9adf54b683147b393258c2c82adc622 (127.0.98.3:46099)
I20260929 17:53:40.469280 8614 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:52631
I20260929 17:53:40.474460 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704375975685-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704375975685-392-0/krb5kdc/oryx_127.0.98.1.keytab.
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704375975685-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704375975685-392-0/krb5kdc/oryx_127.0.98.2.keytab.
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704375975685-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704375975685-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260929 17:53:40.536659 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskup3Soe/build/debug/bin/kudu with pid 8676
I20260929 17:53:40.542407 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskup3Soe/build/debug/bin/kudu with pid 8810
I20260929 17:53:40.547238 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskup3Soe/build/debug/bin/kudu with pid 8944
I20260929 17:53:40.551503 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskup3Soe/build/debug/bin/kudu with pid 8582
I20260929 17:53:40.556694 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskup3Soe/build/debug/bin/kudu
/tmp/dist-test-taskup3Soe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704375975685-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704375975685-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704375975685-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704375975685-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38589
--webserver_interface=127.0.98.62
--webserver_port=41871
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704375975685-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38589
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704375975685-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704375975685-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704375975685-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:53:40.663239 9082 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:53:40.663491 9082 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:53:40.663529 9082 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:53:40.667500 9082 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 17:53:40.667562 9082 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:53:40.667580 9082 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:53:40.667595 9082 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 17:53:40.667608 9082 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 17:53:40.672731 9082 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704375975685-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704375975685-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38589
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704375975685-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38589
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704375975685-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=41871
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9082
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704375975685-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:53:40.673799 9082 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:53:40.674751 9082 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:53:40.682718 9087 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:53:40.682741 9090 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:53:40.682729 9088 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:53:40.682922 9082 server_base.cc:1034] running on GCE node
I20260929 17:53:40.683459 9082 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:53:40.684552 9082 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:53:40.685714 9082 hybrid_clock.cc:648] HybridClock initialized: now 1790704420685696 us; error 31 us; skew 500 ppm
Sep 29 17:53:40 dist-test-slave-53nh krb5kdc[8569](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704420, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:53:40.688301 9082 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 17:53:40.689165 9082 webserver.cc:460] Webserver started at http://127.0.98.62:41871/ using document root <none> and password file <none>
I20260929 17:53:40.689659 9082 fs_manager.cc:359] Metadata directory not provided
I20260929 17:53:40.689724 9082 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:53:40.693344 9082 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260929 17:53:40.695271 9097 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:53:40.696463 9082 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 17:53:40.696557 9082 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704375975685-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704375975685-392-0/minicluster-data/master-0/wal
uuid: "672d23dbfaea49e78e0cc754dbc16953"
format_stamp: "Formatted at 2026-09-29 17:53:39 on dist-test-slave-53nh"
server_key: "1b1552f22a33d2f1459c0298ef2dbe82"
server_key_iv: "a080c65d54bd0cd54361a4cc8e9f1842"
server_key_version: "encryptionkey@0"
I20260929 17:53:40.697096 9082 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704375975685-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704375975685-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704375975685-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:53:40.718014 9082 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:53:40.721225 9082 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:53:40.721402 9082 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:53:40.728235 9082 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38589
I20260929 17:53:40.728255 9149 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38589 every 8 connection(s)
I20260929 17:53:40.729321 9082 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704375975685-392-0/minicluster-data/master-0/data/info.pb
I20260929 17:53:40.732143 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskup3Soe/build/debug/bin/kudu as pid 9082
I20260929 17:53:40.732583 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskup3Soe/build/debug/bin/kudu
/tmp/dist-test-taskup3Soe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704375975685-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704375975685-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704375975685-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704375975685-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:36233
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:38589
--webserver_port=35681
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704375975685-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.1
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704375975685-392-0/krb5kdc/oryx_127.0.98.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704375975685-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704375975685-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704375975685-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 17:53:40.733785 9150 sys_catalog.cc:263] Verifying existing consensus state
I20260929 17:53:40.735340 9150 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 672d23dbfaea49e78e0cc754dbc16953: Bootstrap starting.
I20260929 17:53:40.751137 9150 log.cc:824] T 00000000000000000000000000000000 P 672d23dbfaea49e78e0cc754dbc16953: Log is configured to *not* fsync() on all Append() calls
I20260929 17:53:40.757701 9150 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 672d23dbfaea49e78e0cc754dbc16953: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 17:53:40.758044 9150 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 672d23dbfaea49e78e0cc754dbc16953: Bootstrap complete.
I20260929 17:53:40.762006 9150 raft_consensus.cc:348] T 00000000000000000000000000000000 P 672d23dbfaea49e78e0cc754dbc16953 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "672d23dbfaea49e78e0cc754dbc16953" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38589 } }
I20260929 17:53:40.762379 9150 raft_consensus.cc:729] T 00000000000000000000000000000000 P 672d23dbfaea49e78e0cc754dbc16953 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 672d23dbfaea49e78e0cc754dbc16953, State: Initialized, Role: FOLLOWER
I20260929 17:53:40.762825 9150 consensus_queue.cc:260] T 00000000000000000000000000000000 P 672d23dbfaea49e78e0cc754dbc16953 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "672d23dbfaea49e78e0cc754dbc16953" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38589 } }
I20260929 17:53:40.762934 9150 raft_consensus.cc:388] T 00000000000000000000000000000000 P 672d23dbfaea49e78e0cc754dbc16953 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 17:53:40.762992 9150 raft_consensus.cc:482] T 00000000000000000000000000000000 P 672d23dbfaea49e78e0cc754dbc16953 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 17:53:40.763077 9150 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 672d23dbfaea49e78e0cc754dbc16953 [term 1 FOLLOWER]: Advancing to term 2
I20260929 17:53:40.764904 9150 raft_consensus.cc:504] T 00000000000000000000000000000000 P 672d23dbfaea49e78e0cc754dbc16953 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "672d23dbfaea49e78e0cc754dbc16953" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38589 } }
I20260929 17:53:40.765234 9150 leader_election.cc:302] T 00000000000000000000000000000000 P 672d23dbfaea49e78e0cc754dbc16953 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 672d23dbfaea49e78e0cc754dbc16953; no voters:
I20260929 17:53:40.765512 9150 leader_election.cc:288] T 00000000000000000000000000000000 P 672d23dbfaea49e78e0cc754dbc16953 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260929 17:53:40.765676 9154 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 672d23dbfaea49e78e0cc754dbc16953 [term 2 FOLLOWER]: Leader election won for term 2
I20260929 17:53:40.765906 9154 raft_consensus.cc:686] T 00000000000000000000000000000000 P 672d23dbfaea49e78e0cc754dbc16953 [term 2 LEADER]: Becoming Leader. State: Replica: 672d23dbfaea49e78e0cc754dbc16953, State: Running, Role: LEADER
I20260929 17:53:40.766209 9154 consensus_queue.cc:237] T 00000000000000000000000000000000 P 672d23dbfaea49e78e0cc754dbc16953 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "672d23dbfaea49e78e0cc754dbc16953" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38589 } }
I20260929 17:53:40.766649 9150 sys_catalog.cc:564] T 00000000000000000000000000000000 P 672d23dbfaea49e78e0cc754dbc16953 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 17:53:40.767480 9155 sys_catalog.cc:455] T 00000000000000000000000000000000 P 672d23dbfaea49e78e0cc754dbc16953 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "672d23dbfaea49e78e0cc754dbc16953" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "672d23dbfaea49e78e0cc754dbc16953" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38589 } } }
I20260929 17:53:40.767598 9155 sys_catalog.cc:458] T 00000000000000000000000000000000 P 672d23dbfaea49e78e0cc754dbc16953 [sys.catalog]: This master's current role is: LEADER
I20260929 17:53:40.767894 9156 sys_catalog.cc:455] T 00000000000000000000000000000000 P 672d23dbfaea49e78e0cc754dbc16953 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 672d23dbfaea49e78e0cc754dbc16953. Latest consensus state: current_term: 2 leader_uuid: "672d23dbfaea49e78e0cc754dbc16953" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "672d23dbfaea49e78e0cc754dbc16953" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38589 } } }
I20260929 17:53:40.767971 9156 sys_catalog.cc:458] T 00000000000000000000000000000000 P 672d23dbfaea49e78e0cc754dbc16953 [sys.catalog]: This master's current role is: LEADER
I20260929 17:53:40.771435 9170 catalog_manager.cc:1259] Loaded cluster ID: a5a9de8f68d148e989e5be49b9242f83
I20260929 17:53:40.771490 9170 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 672d23dbfaea49e78e0cc754dbc16953: loading cluster ID for follower catalog manager: success
I20260929 17:53:40.773823 9170 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 672d23dbfaea49e78e0cc754dbc16953: acquiring CA information for follower catalog manager: success
I20260929 17:53:40.775197 9170 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 672d23dbfaea49e78e0cc754dbc16953: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 17:53:40.775341 9163 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 17:53:40.776468 9163 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 17:53:40.776916 9163 catalog_manager.cc:1259] Loaded cluster ID: a5a9de8f68d148e989e5be49b9242f83
I20260929 17:53:40.776970 9163 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 17:53:40.777657 9163 catalog_manager.cc:1504] Loading token signing keys...
I20260929 17:53:40.778122 9163 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 672d23dbfaea49e78e0cc754dbc16953: Loaded TSK: 0
I20260929 17:53:40.778868 9163 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 17:53:40.844039 9152 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:53:40.844321 9152 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:53:40.844381 9152 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:53:40.848389 9152 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:53:40.848452 9152 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:53:40.848534 9152 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 17:53:40.853735 9152 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704375975685-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704375975685-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704375975685-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:36233
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704375975685-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=35681
--tserver_master_addrs=127.0.98.62:38589
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9152
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704375975685-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:53:40.854746 9152 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:53:40.855717 9152 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:53:40.864866 9176 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:53:40.865072 9152 server_base.cc:1034] running on GCE node
W20260929 17:53:40.865161 9179 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:53:40.864945 9177 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:53:40.865597 9152 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:53:40.866232 9152 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:53:40.867427 9152 hybrid_clock.cc:648] HybridClock initialized: now 1790704420867396 us; error 48 us; skew 500 ppm
Sep 29 17:53:40 dist-test-slave-53nh krb5kdc[8569](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704420, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:53:40.870211 9152 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260929 17:53:40.871166 9152 webserver.cc:460] Webserver started at http://127.0.98.1:35681/ using document root <none> and password file <none>
I20260929 17:53:40.871703 9152 fs_manager.cc:359] Metadata directory not provided
I20260929 17:53:40.871785 9152 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:53:40.875684 9152 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260929 17:53:40.877715 9186 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:53:40.878954 9152 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 17:53:40.879065 9152 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704375975685-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704375975685-392-0/minicluster-data/ts-0/wal
uuid: "3c5cb2ae7921438788cb6ee7df2aa560"
format_stamp: "Formatted at 2026-09-29 17:53:40 on dist-test-slave-53nh"
server_key: "46d9ba2c24a2a4aa30943bffc149e6e3"
server_key_iv: "0477e495340560c7092e6e0545534bf3"
server_key_version: "encryptionkey@0"
I20260929 17:53:40.879598 9152 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704375975685-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704375975685-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704375975685-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:53:40.904429 9152 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:53:40.907537 9152 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:53:40.907709 9152 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:53:40.908282 9152 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 17:53:40.909104 9152 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:53:40.909152 9152 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:53:40.909214 9152 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:53:40.909238 9152 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 17:53:40.917600 9152 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36233
I20260929 17:53:40.917631 9299 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36233 every 8 connection(s)
I20260929 17:53:40.918555 9152 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704375975685-392-0/minicluster-data/ts-0/data/info.pb
I20260929 17:53:40.928800 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskup3Soe/build/debug/bin/kudu as pid 9152
Sep 29 17:53:40 dist-test-slave-53nh krb5kdc[8569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 17:53:40.929303 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskup3Soe/build/debug/bin/kudu
/tmp/dist-test-taskup3Soe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704375975685-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704375975685-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704375975685-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704375975685-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:45123
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:38589
--webserver_port=37541
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704375975685-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.2
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704375975685-392-0/krb5kdc/oryx_127.0.98.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704375975685-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704375975685-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704375975685-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 17:53:40 dist-test-slave-53nh krb5kdc[8569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 17:53:40.931267 9303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:40.920661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38589 (local address 127.0.98.1:47009)
0929 17:53:40.921128 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:40.921192 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0929 17:53:40.922392 (+ 1200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:53:40.923526 (+ 1134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:53:40.923535 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:53:40.923956 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:53:40.924413 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:40.924427 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:40.925701 (+ 1274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:40.925704 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:40.926557 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:40.926566 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:40.927272 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:40.927276 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:40.927444 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:40.927965 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:53:40.927980 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:53:40.930458 (+ 2478us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38589: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":122,"threads_started":1}
W20260929 17:53:40.931628 9302 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:53:40.920539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47009 (local address 127.0.98.62:38589)
0929 17:53:40.921022 (+ 483us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:40.921042 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:40.921298 (+ 256us) server_negotiation.cc:407] Connection header received
0929 17:53:40.922801 (+ 1503us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:40.922812 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:40.923110 (+ 298us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:40.923441 (+ 331us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:53:40.924573 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:40.925577 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:40.926676 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:40.927129 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:40.927273 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:40.930719 (+ 3446us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:47009: BlockingRecv error: recv got EOF from 127.0.98.1:47009 (error 108)
Metrics: {"server-negotiator.queue_time_us":301,"thread_start_us":121,"threads_started":1}
W20260929 17:53:40.931720 9300 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:38589 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:38589: Client connection negotiation failed: client connection to 127.0.98.62:38589: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Sep 29 17:53:40 dist-test-slave-53nh krb5kdc[8569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 17:53:40 dist-test-slave-53nh krb5kdc[8569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 17:53:40.937390 9303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:40.932205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38589 (local address 127.0.98.1:39749)
0929 17:53:40.932385 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:40.932399 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 17:53:40.932485 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:53:40.932746 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:53:40.932749 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:53:40.932884 (+ 135us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:53:40.932976 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:40.932982 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:40.933917 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:40.933919 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:40.934465 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:40.934472 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:40.935017 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:40.935019 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:40.935081 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:40.935529 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:53:40.935541 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:53:40.937252 (+ 1711us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38589: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":117}
W20260929 17:53:40.937788 9302 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:53:40.932278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39749 (local address 127.0.98.62:38589)
0929 17:53:40.932384 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:40.932388 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:40.932451 (+ 63us) server_negotiation.cc:407] Connection header received
0929 17:53:40.932572 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:40.932574 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:40.932620 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:40.932714 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:53:40.933102 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:40.933828 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:40.934569 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:40.934915 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:40.935017 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:40.937382 (+ 2365us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:39749: BlockingRecv error: recv got EOF from 127.0.98.1:39749 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Sep 29 17:53:40 dist-test-slave-53nh krb5kdc[8569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 17:53:40 dist-test-slave-53nh krb5kdc[8569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 17:53:40.942469 9303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:40.937941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38589 (local address 127.0.98.1:34199)
0929 17:53:40.938094 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:40.938105 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 17:53:40.938171 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:53:40.938412 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:53:40.938414 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:53:40.938548 (+ 134us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:53:40.938614 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:40.938619 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:40.939293 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:40.939296 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:40.939806 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:40.939813 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:40.940195 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:40.940198 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:40.940267 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:40.940677 (+ 410us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:53:40.940688 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:53:40.942351 (+ 1663us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38589: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":90}
W20260929 17:53:40.942569 9302 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:53:40.938103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34199 (local address 127.0.98.62:38589)
0929 17:53:40.938226 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:40.938230 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:40.938242 (+ 12us) server_negotiation.cc:407] Connection header received
0929 17:53:40.938281 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:40.938283 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:40.938316 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:40.938383 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:53:40.938713 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:40.939194 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:40.939905 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:40.940106 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:40.940187 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:40.942474 (+ 2287us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:34199: BlockingRecv error: recv got EOF from 127.0.98.1:34199 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260929 17:53:40.942688 9300 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 17:53:41.042177 9304 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:53:41.042456 9304 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:53:41.042517 9304 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:53:41.046547 9304 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:53:41.046600 9304 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:53:41.046679 9304 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 17:53:41.051935 9304 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704375975685-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704375975685-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704375975685-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:45123
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704375975685-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=37541
--tserver_master_addrs=127.0.98.62:38589
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9304
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704375975685-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:53:41.052964 9304 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:53:41.054005 9304 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:53:41.063597 9312 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:53:41.063527 9309 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:53:41.063548 9304 server_base.cc:1034] running on GCE node
W20260929 17:53:41.063774 9310 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:53:41.064154 9304 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:53:41.064803 9304 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:53:41.065955 9304 hybrid_clock.cc:648] HybridClock initialized: now 1790704421065933 us; error 33 us; skew 500 ppm
Sep 29 17:53:41 dist-test-slave-53nh krb5kdc[8569](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704421, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:53:41.068769 9304 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260929 17:53:41.069686 9304 webserver.cc:460] Webserver started at http://127.0.98.2:37541/ using document root <none> and password file <none>
I20260929 17:53:41.070248 9304 fs_manager.cc:359] Metadata directory not provided
I20260929 17:53:41.070331 9304 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:53:41.074206 9304 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260929 17:53:41.076198 9319 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:53:41.077489 9304 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 17:53:41.077602 9304 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704375975685-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704375975685-392-0/minicluster-data/ts-1/wal
uuid: "a43890f8f55a40a2ac52fc685c42ed62"
format_stamp: "Formatted at 2026-09-29 17:53:40 on dist-test-slave-53nh"
server_key: "2d5ab5d65eba4670811c7777b9e739d7"
server_key_iv: "af34f9697014f9c407697b948a28d2a8"
server_key_version: "encryptionkey@0"
I20260929 17:53:41.078131 9304 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704375975685-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704375975685-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704375975685-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:53:41.116422 9304 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:53:41.119764 9304 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:53:41.119954 9304 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:53:41.120533 9304 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 17:53:41.121404 9304 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:53:41.121452 9304 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:53:41.121490 9304 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:53:41.121510 9304 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 17:53:41.129743 9304 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45123
I20260929 17:53:41.129768 9432 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45123 every 8 connection(s)
I20260929 17:53:41.130823 9304 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704375975685-392-0/minicluster-data/ts-1/data/info.pb
I20260929 17:53:41.137362 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskup3Soe/build/debug/bin/kudu as pid 9304
I20260929 17:53:41.137820 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskup3Soe/build/debug/bin/kudu
/tmp/dist-test-taskup3Soe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704375975685-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704375975685-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704375975685-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704375975685-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:46099
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:38589
--webserver_port=33999
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704375975685-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.3
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704375975685-392-0/krb5kdc/oryx_127.0.98.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704375975685-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704375975685-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704375975685-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 17:53:41 dist-test-slave-53nh krb5kdc[8569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 17:53:41 dist-test-slave-53nh krb5kdc[8569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260929 17:53:41.141705 9302 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:53:41.132515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51401 (local address 127.0.98.62:38589)
0929 17:53:41.132676 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:41.132681 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:41.133571 (+ 890us) server_negotiation.cc:407] Connection header received
0929 17:53:41.134834 (+ 1263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:41.134836 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:41.134887 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:41.134959 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:53:41.136048 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:41.136575 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:41.137955 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:41.138218 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:41.138307 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:41.141604 (+ 3297us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:51401: BlockingRecv error: recv got EOF from 127.0.98.2:51401 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 17:53:41.142505 9435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:41.132943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38589 (local address 127.0.98.2:51401)
0929 17:53:41.133417 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:41.133459 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 17:53:41.134627 (+ 1168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:53:41.135113 (+ 486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:53:41.135120 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:53:41.135452 (+ 332us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:53:41.135938 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:41.135951 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:41.136672 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:41.136674 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:41.137836 (+ 1162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:41.137847 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:41.138366 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:41.138370 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:41.138556 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:41.139127 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:53:41.139142 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:53:41.141492 (+ 2350us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38589: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":108,"threads_started":1}
W20260929 17:53:41.142997 9433 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:38589 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:38589: Client connection negotiation failed: client connection to 127.0.98.62:38589: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Sep 29 17:53:41 dist-test-slave-53nh krb5kdc[8569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 17:53:41 dist-test-slave-53nh krb5kdc[8569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 17:53:41.148396 9435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:41.143385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38589 (local address 127.0.98.2:38591)
0929 17:53:41.143558 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:41.143570 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 17:53:41.143630 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:53:41.143860 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:53:41.143862 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:53:41.143981 (+ 119us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:53:41.144065 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:41.144070 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:41.145044 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:41.145046 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:41.145574 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:41.145581 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:41.146104 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:41.146107 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:41.146168 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:41.146595 (+ 427us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:53:41.146606 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:53:41.148304 (+ 1698us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38589: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":109}
W20260929 17:53:41.148567 9302 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:53:41.143497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38591 (local address 127.0.98.62:38589)
0929 17:53:41.143616 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:41.143620 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:41.143635 (+ 15us) server_negotiation.cc:407] Connection header received
0929 17:53:41.143705 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:41.143708 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:41.143755 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:41.143853 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:53:41.144160 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:41.144926 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:41.145685 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:41.146010 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:41.146115 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:41.148454 (+ 2339us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:38591: BlockingRecv error: recv got EOF from 127.0.98.2:38591 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Sep 29 17:53:41 dist-test-slave-53nh krb5kdc[8569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 17:53:41 dist-test-slave-53nh krb5kdc[8569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 17:53:41.153717 9435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:41.148926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38589 (local address 127.0.98.2:34189)
0929 17:53:41.149104 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:41.149118 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 17:53:41.149175 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:53:41.149369 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:53:41.149371 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:53:41.149481 (+ 110us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:53:41.149546 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:41.149550 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:41.150475 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:41.150477 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:41.150976 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:41.150982 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:41.151475 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:41.151478 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:41.151549 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:41.151967 (+ 418us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:53:41.151978 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:53:41.153626 (+ 1648us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38589: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":117}
W20260929 17:53:41.153895 9302 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:53:41.149041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34189 (local address 127.0.98.62:38589)
0929 17:53:41.149142 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:41.149146 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:41.149160 (+ 14us) server_negotiation.cc:407] Connection header received
0929 17:53:41.149237 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:41.149240 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:41.149280 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:41.149347 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:53:41.149625 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:41.150393 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:41.151058 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:41.151384 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:41.151487 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:41.153781 (+ 2294us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:34189: BlockingRecv error: recv got EOF from 127.0.98.2:34189 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
W20260929 17:53:41.153997 9433 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 17:53:41.246838 9436 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:53:41.247112 9436 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:53:41.247155 9436 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:53:41.251184 9436 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:53:41.251246 9436 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:53:41.251309 9436 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 17:53:41.256641 9436 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704375975685-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704375975685-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704375975685-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:46099
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704375975685-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=33999
--tserver_master_addrs=127.0.98.62:38589
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9436
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704375975685-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:53:41.257772 9436 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:53:41.258782 9436 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:53:41.267969 9441 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:53:41.268082 9436 server_base.cc:1034] running on GCE node
W20260929 17:53:41.268009 9442 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:53:41.267990 9444 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:53:41.269086 9436 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:53:41.269697 9436 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:53:41.270865 9436 hybrid_clock.cc:648] HybridClock initialized: now 1790704421270833 us; error 47 us; skew 500 ppm
Sep 29 17:53:41 dist-test-slave-53nh krb5kdc[8569](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704421, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:53:41.273590 9436 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260929 17:53:41.274456 9436 webserver.cc:460] Webserver started at http://127.0.98.3:33999/ using document root <none> and password file <none>
I20260929 17:53:41.274996 9436 fs_manager.cc:359] Metadata directory not provided
I20260929 17:53:41.275075 9436 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:53:41.278964 9436 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 17:53:41.280941 9451 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:53:41.282269 9436 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 17:53:41.282372 9436 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704375975685-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704375975685-392-0/minicluster-data/ts-2/wal
uuid: "d9adf54b683147b393258c2c82adc622"
format_stamp: "Formatted at 2026-09-29 17:53:40 on dist-test-slave-53nh"
server_key: "fc0dc88aef97cef4c0bfc0f9e6d7e224"
server_key_iv: "3647b780a2d3e5b845e08a47b145a5bd"
server_key_version: "encryptionkey@0"
I20260929 17:53:41.282944 9436 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704375975685-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704375975685-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704375975685-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:53:41.300489 9436 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:53:41.303726 9436 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:53:41.303921 9436 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:53:41.304514 9436 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 17:53:41.305364 9436 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:53:41.305411 9436 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:53:41.305464 9436 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:53:41.305492 9436 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 17:53:41.313367 9436 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46099
I20260929 17:53:41.313416 9564 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46099 every 8 connection(s)
I20260929 17:53:41.314340 9436 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790704375975685-392-0/minicluster-data/ts-2/data/info.pb
Sep 29 17:53:41 dist-test-slave-53nh krb5kdc[8569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 17:53:41 dist-test-slave-53nh krb5kdc[8569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 17:53:41.324111 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskup3Soe/build/debug/bin/kudu as pid 9436
W20260929 17:53:41.324786 9302 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:53:41.316146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33017 (local address 127.0.98.62:38589)
0929 17:53:41.316275 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:41.316279 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:41.317116 (+ 837us) server_negotiation.cc:407] Connection header received
0929 17:53:41.318398 (+ 1282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:41.318401 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:41.318445 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:41.318515 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:53:41.319717 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:41.320257 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:41.321406 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:41.321622 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:41.321704 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:41.324669 (+ 2965us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:33017: BlockingRecv error: recv got EOF from 127.0.98.3:33017 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 17:53:41.325519 9567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:41.316559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38589 (local address 127.0.98.3:33017)
0929 17:53:41.316974 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:41.317032 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0929 17:53:41.318196 (+ 1164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:53:41.318679 (+ 483us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:53:41.318686 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:53:41.319026 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:53:41.319536 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:41.319551 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:41.320364 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:41.320367 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:41.321301 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:41.321310 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:41.321747 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:41.321750 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:41.321882 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:41.322389 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:53:41.322404 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:53:41.324549 (+ 2145us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38589: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":103,"threads_started":1}
W20260929 17:53:41.325996 9565 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:38589 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:38589: Client connection negotiation failed: client connection to 127.0.98.62:38589: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Sep 29 17:53:41 dist-test-slave-53nh krb5kdc[8569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 17:53:41 dist-test-slave-53nh krb5kdc[8569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 17:53:41.331399 9567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:41.326419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38589 (local address 127.0.98.3:51951)
0929 17:53:41.326504 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:41.326517 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 17:53:41.326581 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:53:41.326880 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:53:41.326882 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:53:41.327021 (+ 139us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:53:41.327111 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:41.327116 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:41.327814 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:41.327817 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:41.328477 (+ 660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:41.328486 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:41.328896 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:41.328899 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:41.328980 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:41.329406 (+ 426us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:53:41.329419 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:53:41.331285 (+ 1866us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38589: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":26}
W20260929 17:53:41.331521 9569 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:53:41.326495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51951 (local address 127.0.98.62:38589)
0929 17:53:41.326713 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:41.326716 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:41.326725 (+ 9us) server_negotiation.cc:407] Connection header received
0929 17:53:41.326750 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:41.326752 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:41.326783 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:41.326854 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:53:41.327211 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:41.327720 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:41.328588 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:41.328807 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:41.328886 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:41.331400 (+ 2514us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:51951: BlockingRecv error: recv got EOF from 127.0.98.3:51951 (error 108)
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":79,"threads_started":1}
I20260929 17:53:41.333252 9302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:41.325044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42346 (local address 127.0.98.62:38589)
0929 17:53:41.325196 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:41.325199 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:41.325211 (+ 12us) server_negotiation.cc:407] Connection header received
0929 17:53:41.325354 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:41.325356 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:41.325397 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:41.325466 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:53:41.326217 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:41.326772 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:41.327573 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:41.327779 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:41.327858 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:41.329124 (+ 1266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:53:41.329129 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:53:41.329144 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:53:41.329169 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:53:41.330863 (+ 1694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:53:41.331304 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:53:41.331310 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:53:41.331316 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:53:41.331467 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:53:41.331977 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:53:41.331981 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:53:41.331982 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:53:41.332470 (+ 488us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:53:41.332670 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0929 17:53:41.332879 (+ 209us) server_negotiation.cc:299] Negotiation successful
0929 17:53:41.333109 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
Sep 29 17:53:41 dist-test-slave-53nh krb5kdc[8569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 17:53:41 dist-test-slave-53nh krb5kdc[8569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 17:53:41.336642 9567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:41.331973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38589 (local address 127.0.98.3:46881)
0929 17:53:41.332119 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:41.332132 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 17:53:41.332198 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:53:41.332385 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:53:41.332388 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:53:41.332528 (+ 140us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:53:41.332608 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:41.332613 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:41.333467 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:41.333470 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:41.333971 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:41.333978 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:41.334450 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:41.334453 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:41.334526 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:41.334923 (+ 397us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:53:41.334937 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:53:41.336557 (+ 1620us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38589: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":89}
W20260929 17:53:41.336751 9569 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:53:41.332043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46881 (local address 127.0.98.62:38589)
0929 17:53:41.332160 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:41.332162 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:41.332173 (+ 11us) server_negotiation.cc:407] Connection header received
0929 17:53:41.332267 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:41.332269 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:41.332300 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:41.332358 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:53:41.332725 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:41.333374 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:41.334073 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:41.334345 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:41.334420 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:41.336668 (+ 2248us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:46881: BlockingRecv error: recv got EOF from 127.0.98.3:46881 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20260929 17:53:41.336834 9565 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 29 17:53:41 dist-test-slave-53nh krb5kdc[8569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 17:53:41 dist-test-slave-53nh krb5kdc[8569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 17:53:41.949484 9570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:41.943364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38589 (local address 127.0.98.1:49063)
0929 17:53:41.943659 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:41.943675 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 17:53:41.943797 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:53:41.944333 (+ 536us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:53:41.944336 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:53:41.944547 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:53:41.944710 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:41.944718 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:41.945645 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:41.945648 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:41.946215 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:41.946223 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:41.946677 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:41.946681 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:41.946777 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:41.947299 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:53:41.947314 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:53:41.949322 (+ 2008us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38589: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":79,"threads_started":1}
W20260929 17:53:41.949556 9571 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:53:41.943435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49063 (local address 127.0.98.62:38589)
0929 17:53:41.944036 (+ 601us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:41.944040 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:41.944053 (+ 13us) server_negotiation.cc:407] Connection header received
0929 17:53:41.944092 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:41.944096 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:41.944143 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:41.944270 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:53:41.944831 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:41.945537 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:41.946328 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:41.946559 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:41.946658 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:41.949444 (+ 2786us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:49063: BlockingRecv error: recv got EOF from 127.0.98.1:49063 (error 108)
Metrics: {"server-negotiator.queue_time_us":525,"thread_start_us":73,"threads_started":1}
Sep 29 17:53:42 dist-test-slave-53nh krb5kdc[8569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 17:53:42 dist-test-slave-53nh krb5kdc[8569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 17:53:42.159986 9572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:42.154571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38589 (local address 127.0.98.2:32791)
0929 17:53:42.154849 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:42.154863 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 17:53:42.154977 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:53:42.155219 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:53:42.155221 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:53:42.155409 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:53:42.155526 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:42.155532 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:42.156283 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:42.156286 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:42.156856 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:42.156864 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:42.157346 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:42.157350 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:42.157443 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:42.157931 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:53:42.157943 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:53:42.159823 (+ 1880us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38589: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":84,"threads_started":1}
W20260929 17:53:42.160080 9571 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:53:42.154670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:32791 (local address 127.0.98.62:38589)
0929 17:53:42.154812 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:42.154817 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:42.154933 (+ 116us) server_negotiation.cc:407] Connection header received
0929 17:53:42.155064 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:42.155067 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:42.155109 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:42.155177 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:53:42.155635 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:42.156149 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:42.156969 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:42.157236 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:42.157333 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:42.159969 (+ 2636us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:32791: BlockingRecv error: recv got EOF from 127.0.98.2:32791 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 29 17:53:42 dist-test-slave-53nh krb5kdc[8569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 17:53:42 dist-test-slave-53nh krb5kdc[8569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 17:53:42.342336 9573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:42.337414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38589 (local address 127.0.98.3:47553)
0929 17:53:42.337645 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:42.337659 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 17:53:42.337767 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:53:42.337972 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:53:42.337974 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:53:42.338157 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:53:42.338268 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:42.338274 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:42.338966 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:42.338968 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:42.339510 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:42.339517 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:42.339969 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:42.339971 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:42.340039 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:42.340501 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:53:42.340512 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:53:42.342204 (+ 1692us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38589: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":64,"threads_started":1}
W20260929 17:53:42.342437 9571 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:53:42.337491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47553 (local address 127.0.98.62:38589)
0929 17:53:42.337642 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:42.337646 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:42.337721 (+ 75us) server_negotiation.cc:407] Connection header received
0929 17:53:42.337838 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:42.337840 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:42.337884 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:42.337947 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:53:42.338363 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:42.338891 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:42.339638 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:42.339871 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:42.339965 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:42.342342 (+ 2377us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:47553: BlockingRecv error: recv got EOF from 127.0.98.3:47553 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Sep 29 17:53:42 dist-test-slave-53nh krb5kdc[8569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 17:53:42 dist-test-slave-53nh krb5kdc[8569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 17:53:42.955641 9574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:42.950470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38589 (local address 127.0.98.1:52193)
0929 17:53:42.950760 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:42.950777 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 17:53:42.950880 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:53:42.951149 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:53:42.951152 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:53:42.951334 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:53:42.951451 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:42.951459 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:42.952283 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:42.952286 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:42.952841 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:42.952850 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:42.953241 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:42.953244 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:42.953305 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:42.953776 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:53:42.953788 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:53:42.955475 (+ 1687us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38589: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":85,"threads_started":1}
W20260929 17:53:42.955736 9575 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:53:42.950613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52193 (local address 127.0.98.62:38589)
0929 17:53:42.950822 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:42.950826 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:42.950839 (+ 13us) server_negotiation.cc:407] Connection header received
0929 17:53:42.950970 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:42.950972 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:42.951021 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:42.951104 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:53:42.951562 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:42.952186 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:42.952930 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:42.953156 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:42.953222 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:42.955613 (+ 2391us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:52193: BlockingRecv error: recv got EOF from 127.0.98.1:52193 (error 108)
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":76,"threads_started":1}
Sep 29 17:53:43 dist-test-slave-53nh krb5kdc[8569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 17:53:43 dist-test-slave-53nh krb5kdc[8569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 17:53:43.165553 9576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:43.160670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38589 (local address 127.0.98.2:59281)
0929 17:53:43.160930 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:43.160944 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 17:53:43.161091 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:53:43.161302 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:53:43.161304 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:53:43.161476 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:53:43.161571 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:43.161577 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:43.162289 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:43.162291 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:43.162839 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:43.162845 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:43.163283 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:43.163285 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:43.163341 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:43.163797 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:53:43.163808 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:53:43.165423 (+ 1615us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38589: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":90,"threads_started":1}
W20260929 17:53:43.165644 9575 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:53:43.160762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59281 (local address 127.0.98.62:38589)
0929 17:53:43.160888 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:43.160892 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:43.161040 (+ 148us) server_negotiation.cc:407] Connection header received
0929 17:53:43.161160 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:43.161162 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:43.161209 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:43.161279 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:53:43.161658 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:43.162205 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:43.162945 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:43.163197 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:43.163279 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:43.165541 (+ 2262us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:59281: BlockingRecv error: recv got EOF from 127.0.98.2:59281 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Sep 29 17:53:43 dist-test-slave-53nh krb5kdc[8569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 17:53:43 dist-test-slave-53nh krb5kdc[8569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 17:53:43.348208 9577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:43.343085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38589 (local address 127.0.98.3:54675)
0929 17:53:43.343371 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:43.343388 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 17:53:43.343494 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:53:43.343748 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:53:43.343751 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:53:43.343975 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:53:43.344099 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:43.344107 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:43.344848 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:43.344852 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:43.345464 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:43.345473 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:43.345886 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:43.345889 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:43.345960 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:43.346404 (+ 444us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:53:43.346414 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:53:43.348030 (+ 1616us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38589: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":96,"threads_started":1}
W20260929 17:53:43.348284 9575 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:53:43.343278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54675 (local address 127.0.98.62:38589)
0929 17:53:43.343414 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:43.343419 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:43.343437 (+ 18us) server_negotiation.cc:407] Connection header received
0929 17:53:43.343586 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:43.343589 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:43.343636 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:43.343713 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:53:43.344225 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:43.344754 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:43.345569 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:43.345795 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:43.345877 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:43.348161 (+ 2284us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:54675: BlockingRecv error: recv got EOF from 127.0.98.3:54675 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 29 17:53:43 dist-test-slave-53nh krb5kdc[8569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 17:53:43 dist-test-slave-53nh krb5kdc[8569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 17:53:43.961962 9578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:43.956484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38589 (local address 127.0.98.1:36221)
0929 17:53:43.956774 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:43.956790 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 17:53:43.956907 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:53:43.957228 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:53:43.957231 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:53:43.957428 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:53:43.957524 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:43.957530 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:43.958290 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:43.958293 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:43.958875 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:43.958883 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:43.959337 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:43.959341 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:43.959420 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:43.959958 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:53:43.959970 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:53:43.961791 (+ 1821us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38589: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":83,"threads_started":1}
W20260929 17:53:43.962100 9579 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:53:43.956534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36221 (local address 127.0.98.62:38589)
0929 17:53:43.956803 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:43.956808 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:43.956851 (+ 43us) server_negotiation.cc:407] Connection header received
0929 17:53:43.957014 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:43.957018 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:43.957105 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:43.957187 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:53:43.957646 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:43.958190 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:43.958992 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:43.959227 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:43.959316 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:43.961941 (+ 2625us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:36221: BlockingRecv error: recv got EOF from 127.0.98.1:36221 (error 108)
Metrics: {"server-negotiator.queue_time_us":207,"thread_start_us":95,"threads_started":1}
Sep 29 17:53:44 dist-test-slave-53nh krb5kdc[8569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 17:53:44 dist-test-slave-53nh krb5kdc[8569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 17:53:44.171810 9580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:44.166361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38589 (local address 127.0.98.2:37257)
0929 17:53:44.166701 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:44.166716 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 17:53:44.166829 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:53:44.167076 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:53:44.167079 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:53:44.167292 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:53:44.167411 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:44.167420 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:44.168322 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:44.168326 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:44.168894 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:44.168903 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:44.169356 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:44.169360 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:44.169433 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:44.169934 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:53:44.169946 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:53:44.171641 (+ 1695us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38589: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":78,"threads_started":1}
W20260929 17:53:44.171900 9579 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:53:44.166432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37257 (local address 127.0.98.62:38589)
0929 17:53:44.166558 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:44.166562 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:44.166776 (+ 214us) server_negotiation.cc:407] Connection header received
0929 17:53:44.166914 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:44.166917 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:44.166968 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:44.167039 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:53:44.167520 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:44.168144 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:44.168992 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:44.169228 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:44.169302 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:44.171801 (+ 2499us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:37257: BlockingRecv error: recv got EOF from 127.0.98.2:37257 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Sep 29 17:53:44 dist-test-slave-53nh krb5kdc[8569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 17:53:44 dist-test-slave-53nh krb5kdc[8569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 17:53:44.354476 9581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:44.349117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38589 (local address 127.0.98.3:59321)
0929 17:53:44.349374 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:44.349385 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 17:53:44.349491 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:53:44.349748 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:53:44.349751 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:53:44.349942 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:53:44.350029 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:44.350035 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:44.350766 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:44.350769 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:44.351326 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:44.351333 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:44.351759 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:44.351763 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:44.351838 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:44.352322 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:53:44.352334 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:53:44.354344 (+ 2010us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38589: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":95,"threads_started":1}
W20260929 17:53:44.354687 9579 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:53:44.349146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59321 (local address 127.0.98.62:38589)
0929 17:53:44.349302 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:44.349307 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:44.349453 (+ 146us) server_negotiation.cc:407] Connection header received
0929 17:53:44.349579 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:44.349582 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:44.349632 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:44.349714 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:53:44.350141 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:44.350666 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:44.351431 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:44.351663 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:44.351747 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:44.354546 (+ 2799us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:59321: BlockingRecv error: recv got EOF from 127.0.98.3:59321 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Sep 29 17:53:44 dist-test-slave-53nh krb5kdc[8569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 17:53:44 dist-test-slave-53nh krb5kdc[8569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 17:53:44.975113 9582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:44.962697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38589 (local address 127.0.98.1:38763)
0929 17:53:44.962984 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:44.963001 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 17:53:44.963107 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:53:44.963364 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:53:44.963367 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:53:44.963554 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:53:44.963663 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:44.963670 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:44.964501 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:44.964504 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:44.965187 (+ 683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:44.965196 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:44.965631 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:44.965640 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:44.965710 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:44.973083 (+ 7373us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:53:44.973102 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:53:44.974950 (+ 1848us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38589: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":83,"threads_started":1}
W20260929 17:53:44.975217 9583 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:53:44.962838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38763 (local address 127.0.98.62:38589)
0929 17:53:44.963080 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:44.963083 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:44.963096 (+ 13us) server_negotiation.cc:407] Connection header received
0929 17:53:44.963190 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:44.963192 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:44.963234 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:44.963306 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:53:44.963820 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:44.964393 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:44.965287 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:44.965511 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:44.965586 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:44.975096 (+ 9510us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:38763: BlockingRecv error: recv got EOF from 127.0.98.1:38763 (error 108)
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":90,"threads_started":1}
Sep 29 17:53:45 dist-test-slave-53nh krb5kdc[8569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 17:53:45 dist-test-slave-53nh krb5kdc[8569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 17:53:45.177709 9584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:45.172606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38589 (local address 127.0.98.2:59787)
0929 17:53:45.172871 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:45.172883 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 17:53:45.172978 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:53:45.173236 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:53:45.173238 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:53:45.173407 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:53:45.173502 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:45.173508 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:45.174231 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:45.174233 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:45.174797 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:45.174803 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:45.175243 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:45.175245 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:45.175304 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:45.175792 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:53:45.175804 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:53:45.177565 (+ 1761us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38589: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":94,"threads_started":1}
W20260929 17:53:45.177793 9583 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:53:45.172692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59787 (local address 127.0.98.62:38589)
0929 17:53:45.172827 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:45.172831 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:45.172946 (+ 115us) server_negotiation.cc:407] Connection header received
0929 17:53:45.173071 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:45.173073 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:45.173117 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:45.173184 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:53:45.173611 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:45.174148 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:45.174917 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:45.175143 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:45.175226 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:45.177698 (+ 2472us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:59787: BlockingRecv error: recv got EOF from 127.0.98.2:59787 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 29 17:53:45 dist-test-slave-53nh krb5kdc[8569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 17:53:45 dist-test-slave-53nh krb5kdc[8569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 17:53:45.360517 9585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:45.355226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38589 (local address 127.0.98.3:42463)
0929 17:53:45.355470 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:45.355483 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 17:53:45.355578 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:53:45.355796 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:53:45.355798 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:53:45.355964 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:53:45.356067 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:45.356071 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:45.356790 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:45.356793 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:45.357528 (+ 735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:45.357540 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:45.358053 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:45.358057 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:45.358122 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:45.358678 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:53:45.358693 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:53:45.360390 (+ 1697us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38589: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":71,"threads_started":1}
W20260929 17:53:45.360596 9583 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:53:45.355313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42463 (local address 127.0.98.62:38589)
0929 17:53:45.355430 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:45.355433 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:45.355543 (+ 110us) server_negotiation.cc:407] Connection header received
0929 17:53:45.355649 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:45.355651 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:45.355698 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:45.355767 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:53:45.356154 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:45.356686 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:45.357644 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:45.357944 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:45.358033 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:45.360503 (+ 2470us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:42463: BlockingRecv error: recv got EOF from 127.0.98.3:42463 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Sep 29 17:53:45 dist-test-slave-53nh krb5kdc[8569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 17:53:45 dist-test-slave-53nh krb5kdc[8569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 17:53:45.981846 9586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:45.975982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38589 (local address 127.0.98.1:36993)
0929 17:53:45.976254 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:45.976269 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 17:53:45.976354 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:53:45.976578 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:53:45.976580 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:53:45.976756 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:53:45.976851 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:45.976856 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:45.977690 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:45.977694 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:45.978469 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:45.978481 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:45.978966 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:45.978969 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:45.979038 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:45.979662 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:53:45.979675 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:53:45.981684 (+ 2009us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38589: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":83,"threads_started":1}
W20260929 17:53:45.981994 9587 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:53:45.976114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36993 (local address 127.0.98.62:38589)
0929 17:53:45.976334 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:45.976337 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:45.976352 (+ 15us) server_negotiation.cc:407] Connection header received
0929 17:53:45.976423 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:45.976425 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:45.976466 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:45.976555 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:53:45.976946 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:45.977566 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:45.978604 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:45.978851 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:45.978936 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:45.981823 (+ 2887us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:36993: BlockingRecv error: recv got EOF from 127.0.98.1:36993 (error 108)
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":67,"threads_started":1}
Sep 29 17:53:46 dist-test-slave-53nh krb5kdc[8569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 17:53:46 dist-test-slave-53nh krb5kdc[8569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 17:53:46.183840 9588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:46.178440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38589 (local address 127.0.98.2:36603)
0929 17:53:46.178758 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:46.178771 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 17:53:46.178862 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:53:46.179120 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:53:46.179122 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:53:46.179317 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:53:46.179413 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:46.179418 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:46.180179 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:46.180182 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:46.180757 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:46.180766 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:46.181294 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:46.181296 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:46.181361 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:46.181800 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:53:46.181810 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:53:46.183628 (+ 1818us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38589: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":108,"threads_started":1}
W20260929 17:53:46.183912 9587 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:53:46.178637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36603 (local address 127.0.98.62:38589)
0929 17:53:46.178773 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:46.178777 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:46.178820 (+ 43us) server_negotiation.cc:407] Connection header received
0929 17:53:46.178939 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:46.178941 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:46.178985 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:46.179068 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:53:46.179504 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:46.180062 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:46.180899 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:46.181190 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:46.181266 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:46.183804 (+ 2538us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:36603: BlockingRecv error: recv got EOF from 127.0.98.2:36603 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 29 17:53:46 dist-test-slave-53nh krb5kdc[8569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 17:53:46 dist-test-slave-53nh krb5kdc[8569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 17:53:46.367018 9589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:46.361420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38589 (local address 127.0.98.3:47353)
0929 17:53:46.361677 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:46.361693 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 17:53:46.361798 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:53:46.362083 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:53:46.362086 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:53:46.362271 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:53:46.362385 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:46.362391 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:46.363138 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:46.363141 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:46.363846 (+ 705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:46.363858 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:46.364339 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:46.364342 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:46.364406 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:46.364970 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:53:46.364985 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:53:46.366876 (+ 1891us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38589: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":79,"threads_started":1}
W20260929 17:53:46.367164 9587 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:53:46.361517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47353 (local address 127.0.98.62:38589)
0929 17:53:46.361649 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:46.361653 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:46.361760 (+ 107us) server_negotiation.cc:407] Connection header received
0929 17:53:46.361900 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:46.361902 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:46.361954 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:46.362014 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:53:46.362487 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:46.363010 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:46.363966 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:46.364226 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:46.364323 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:46.367045 (+ 2722us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:47353: BlockingRecv error: recv got EOF from 127.0.98.3:47353 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 29 17:53:46 dist-test-slave-53nh krb5kdc[8569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 17:53:46 dist-test-slave-53nh krb5kdc[8569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 17:53:46.987741 9590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:46.982602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38589 (local address 127.0.98.1:35253)
0929 17:53:46.982865 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:46.982878 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 17:53:46.982967 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:53:46.983187 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:53:46.983189 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:53:46.983351 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:53:46.983443 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:46.983448 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:46.984246 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:46.984250 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:46.984812 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:46.984820 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:46.985234 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:46.985237 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:46.985297 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:46.985822 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:53:46.985832 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:53:46.987569 (+ 1737us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38589: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":81,"threads_started":1}
W20260929 17:53:46.987823 9591 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:53:46.982689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35253 (local address 127.0.98.62:38589)
0929 17:53:46.982920 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:46.982923 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:46.982936 (+ 13us) server_negotiation.cc:407] Connection header received
0929 17:53:46.983049 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:46.983051 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:46.983093 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:46.983172 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:53:46.983517 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:46.984121 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:46.984898 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:46.985131 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:46.985218 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:46.987694 (+ 2476us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:35253: BlockingRecv error: recv got EOF from 127.0.98.1:35253 (error 108)
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":80,"threads_started":1}
Sep 29 17:53:47 dist-test-slave-53nh krb5kdc[8569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 17:53:47 dist-test-slave-53nh krb5kdc[8569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 17:53:47.190119 9592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:47.184703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38589 (local address 127.0.98.2:58415)
0929 17:53:47.184949 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:47.184964 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 17:53:47.185100 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:53:47.185386 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:53:47.185389 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:53:47.185616 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:53:47.185733 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:47.185742 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:47.186623 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:47.186626 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:47.187260 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:47.187272 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:47.187704 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:47.187707 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:47.187774 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:47.188253 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:53:47.188264 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:53:47.190004 (+ 1740us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38589: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":82,"threads_started":1}
W20260929 17:53:47.190289 9591 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:53:47.184808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58415 (local address 127.0.98.62:38589)
0929 17:53:47.184939 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:47.184942 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:47.185045 (+ 103us) server_negotiation.cc:407] Connection header received
0929 17:53:47.185209 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:47.185212 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:47.185263 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:47.185342 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:53:47.185855 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:47.186525 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:47.187365 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:47.187580 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:47.187656 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:47.190176 (+ 2520us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:58415: BlockingRecv error: recv got EOF from 127.0.98.2:58415 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 29 17:53:47 dist-test-slave-53nh krb5kdc[8569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 17:53:47 dist-test-slave-53nh krb5kdc[8569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 17:53:47.373103 9593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:47.367856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38589 (local address 127.0.98.3:33135)
0929 17:53:47.368108 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:47.368122 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 17:53:47.368221 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:53:47.368449 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:53:47.368452 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:53:47.368681 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:53:47.368774 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:47.368779 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:47.369576 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:47.369580 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:47.370134 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:47.370141 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:47.370566 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:47.370570 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:47.370644 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:47.371140 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:53:47.371154 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:53:47.372926 (+ 1772us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38589: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":79,"threads_started":1}
W20260929 17:53:47.373178 9591 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:53:47.367910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33135 (local address 127.0.98.62:38589)
0929 17:53:47.368039 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:47.368043 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:47.368199 (+ 156us) server_negotiation.cc:407] Connection header received
0929 17:53:47.368290 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:47.368293 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:47.368339 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:47.368408 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:53:47.368891 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:47.369478 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:47.370249 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:47.370474 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:47.370555 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:47.373075 (+ 2520us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:33135: BlockingRecv error: recv got EOF from 127.0.98.3:33135 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 29 17:53:47 dist-test-slave-53nh krb5kdc[8569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 17:53:47 dist-test-slave-53nh krb5kdc[8569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 17:53:47.993705 9594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:47.988540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38589 (local address 127.0.98.1:46145)
0929 17:53:47.988861 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:47.988873 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 17:53:47.988970 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:53:47.989213 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:53:47.989215 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:53:47.989402 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:53:47.989500 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:47.989506 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:47.990270 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:47.990273 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:47.990820 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:47.990828 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:47.991320 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:47.991322 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:47.991385 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:47.991837 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:53:47.991847 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:53:47.993558 (+ 1711us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38589: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":72,"threads_started":1}
W20260929 17:53:47.993853 9595 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:53:47.988632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46145 (local address 127.0.98.62:38589)
0929 17:53:47.988869 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:47.988872 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:47.988937 (+ 65us) server_negotiation.cc:407] Connection header received
0929 17:53:47.989066 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:47.989068 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:47.989114 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:47.989188 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:53:47.989627 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:47.990166 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:47.990937 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:47.991210 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:47.991301 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:47.993707 (+ 2406us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:46145: BlockingRecv error: recv got EOF from 127.0.98.1:46145 (error 108)
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":84,"threads_started":1}
Sep 29 17:53:48 dist-test-slave-53nh krb5kdc[8569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 17:53:48 dist-test-slave-53nh krb5kdc[8569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 17:53:48.196025 9596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:48.190870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38589 (local address 127.0.98.2:47689)
0929 17:53:48.191193 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:48.191208 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 17:53:48.191328 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:53:48.191558 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:53:48.191561 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:53:48.191762 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:53:48.191848 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:48.191854 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:48.192590 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:48.192594 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:48.193193 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:48.193201 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:48.193615 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:48.193618 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:48.193694 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:48.194158 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:53:48.194170 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:53:48.195870 (+ 1700us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38589: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":74,"threads_started":1}
W20260929 17:53:48.196101 9595 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:53:48.190941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47689 (local address 127.0.98.62:38589)
0929 17:53:48.191080 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:48.191084 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:48.191278 (+ 194us) server_negotiation.cc:407] Connection header received
0929 17:53:48.191413 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:48.191416 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:48.191463 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:48.191525 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:53:48.191955 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:48.192489 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:48.193301 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:48.193515 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:48.193603 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:48.196004 (+ 2401us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:47689: BlockingRecv error: recv got EOF from 127.0.98.2:47689 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 29 17:53:48 dist-test-slave-53nh krb5kdc[8569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 17:53:48 dist-test-slave-53nh krb5kdc[8569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 17:53:48.379156 9597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:48.373965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38589 (local address 127.0.98.3:56905)
0929 17:53:48.374326 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:48.374342 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 17:53:48.374461 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:53:48.374725 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:53:48.374728 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:53:48.374947 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:53:48.375059 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:48.375067 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:48.375881 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:48.375883 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:48.376428 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:48.376436 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:48.376811 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:48.376813 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:48.376870 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:48.377345 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:53:48.377356 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:53:48.379013 (+ 1657us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38589: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":76,"threads_started":1}
W20260929 17:53:48.379256 9595 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:53:48.374043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56905 (local address 127.0.98.62:38589)
0929 17:53:48.374197 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:48.374201 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:48.374421 (+ 220us) server_negotiation.cc:407] Connection header received
0929 17:53:48.374546 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:48.374549 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:48.374607 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:48.374689 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:53:48.375174 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:48.375788 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:48.376525 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:48.376735 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:48.376803 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:48.379159 (+ 2356us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:56905: BlockingRecv error: recv got EOF from 127.0.98.3:56905 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 29 17:53:48 dist-test-slave-53nh krb5kdc[8569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 17:53:48 dist-test-slave-53nh krb5kdc[8569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 17:53:48.999786 9598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:48.994500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38589 (local address 127.0.98.1:40645)
0929 17:53:48.994741 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:48.994756 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 17:53:48.994846 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:53:48.995086 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:53:48.995088 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:53:48.995270 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:53:48.995361 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:48.995366 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:48.996075 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:48.996077 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:48.996647 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:48.996656 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:48.997178 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:48.997181 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:48.997254 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:48.997783 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:53:48.997795 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:53:48.999630 (+ 1835us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38589: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":89,"threads_started":1}
W20260929 17:53:48.999924 9599 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:53:48.994671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40645 (local address 127.0.98.62:38589)
0929 17:53:48.994884 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:48.994887 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:48.994901 (+ 14us) server_negotiation.cc:407] Connection header received
0929 17:53:48.994938 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:48.994940 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:48.994980 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:48.995058 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:53:48.995459 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:48.995994 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:48.996781 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:48.997055 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:48.997150 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:48.999792 (+ 2642us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:40645: BlockingRecv error: recv got EOF from 127.0.98.1:40645 (error 108)
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":63,"threads_started":1}
Sep 29 17:53:49 dist-test-slave-53nh krb5kdc[8569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 17:53:49 dist-test-slave-53nh krb5kdc[8569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 17:53:49.202688 9600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:49.196865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38589 (local address 127.0.98.2:43009)
0929 17:53:49.197189 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:49.197205 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 17:53:49.197328 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:53:49.197571 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:53:49.197574 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:53:49.197793 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:53:49.197890 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:49.197896 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:49.198833 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:49.198836 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:49.199416 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:49.199425 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:49.199904 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:49.199908 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:49.199988 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:49.200499 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:53:49.200511 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:53:49.202501 (+ 1990us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38589: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":131,"threads_started":1}
W20260929 17:53:49.202775 9599 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:53:49.197033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43009 (local address 127.0.98.62:38589)
0929 17:53:49.197192 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:49.197196 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:49.197277 (+ 81us) server_negotiation.cc:407] Connection header received
0929 17:53:49.197421 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:49.197424 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:49.197468 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:49.197538 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:53:49.198016 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:49.198717 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:49.199536 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:49.199793 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:49.199886 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:49.202671 (+ 2785us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:43009: BlockingRecv error: recv got EOF from 127.0.98.2:43009 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 29 17:53:49 dist-test-slave-53nh krb5kdc[8569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 17:53:49 dist-test-slave-53nh krb5kdc[8569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 17:53:49.384829 9601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:49.379834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38589 (local address 127.0.98.3:36269)
0929 17:53:49.380159 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:49.380172 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 17:53:49.380270 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:53:49.380475 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:53:49.380477 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:53:49.380644 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:53:49.380737 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:49.380742 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:49.381467 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:49.381470 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:49.382064 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:49.382073 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:49.382507 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:49.382510 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:49.382571 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:49.383029 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:53:49.383039 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:53:49.384717 (+ 1678us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38589: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":92,"threads_started":1}
W20260929 17:53:49.384956 9599 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:53:49.379925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36269 (local address 127.0.98.62:38589)
0929 17:53:49.380053 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:49.380058 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:49.380238 (+ 180us) server_negotiation.cc:407] Connection header received
0929 17:53:49.380347 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:49.380349 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:49.380395 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:49.380448 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:53:49.380817 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:49.381377 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:49.382167 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:49.382408 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:49.382498 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:49.384840 (+ 2342us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:36269: BlockingRecv error: recv got EOF from 127.0.98.3:36269 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 29 17:53:50 dist-test-slave-53nh krb5kdc[8569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 17:53:50 dist-test-slave-53nh krb5kdc[8569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 17:53:50.012531 9602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:50.000528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38589 (local address 127.0.98.1:58347)
0929 17:53:50.000846 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:50.000861 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 17:53:50.000975 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:53:50.001305 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:53:50.001308 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:53:50.001481 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:53:50.001602 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:50.001609 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:50.002394 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:50.002397 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:50.002970 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:50.002978 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:50.003479 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:50.003482 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:50.003554 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:50.010453 (+ 6899us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:53:50.010467 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:53:50.012408 (+ 1941us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38589: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":90,"threads_started":1}
W20260929 17:53:50.012732 9603 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:53:50.000624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58347 (local address 127.0.98.62:38589)
0929 17:53:50.000872 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:50.000875 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:50.000933 (+ 58us) server_negotiation.cc:407] Connection header received
0929 17:53:50.001130 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:50.001133 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:50.001180 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:50.001257 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:53:50.001704 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:50.002289 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:50.003125 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:50.003355 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:50.003436 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:50.012589 (+ 9153us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:58347: BlockingRecv error: recv got EOF from 127.0.98.1:58347 (error 108)
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":60,"threads_started":1}
Sep 29 17:53:50 dist-test-slave-53nh krb5kdc[8569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 17:53:50 dist-test-slave-53nh krb5kdc[8569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 17:53:50.209244 9604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:50.203467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38589 (local address 127.0.98.2:33303)
0929 17:53:50.203787 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:50.203802 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 17:53:50.203913 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:53:50.204178 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:53:50.204181 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:53:50.204354 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:53:50.204450 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:50.204455 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:50.205272 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:50.205275 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:50.205834 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:50.205842 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:50.206264 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:50.206266 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:50.206321 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:50.206875 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:53:50.206887 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:53:50.209084 (+ 2197us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38589: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":115,"threads_started":1}
W20260929 17:53:50.209296 9603 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:53:50.203614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33303 (local address 127.0.98.62:38589)
0929 17:53:50.203790 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:50.203795 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:50.203912 (+ 117us) server_negotiation.cc:407] Connection header received
0929 17:53:50.203979 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:50.203982 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:50.204030 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:50.204103 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:53:50.204579 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:50.205179 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:50.205932 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:50.206178 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:50.206255 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:50.209207 (+ 2952us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:33303: BlockingRecv error: recv got EOF from 127.0.98.2:33303 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Sep 29 17:53:50 dist-test-slave-53nh krb5kdc[8569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 17:53:50 dist-test-slave-53nh krb5kdc[8569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 17:53:50.390821 9605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:50.385518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38589 (local address 127.0.98.3:47609)
0929 17:53:50.385793 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:50.385806 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 17:53:50.385901 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:53:50.386112 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:53:50.386114 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:53:50.386305 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:53:50.386396 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:50.386401 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:50.387256 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:50.387259 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:50.387859 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:50.387869 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:50.388290 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:50.388292 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:50.388347 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:50.388840 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:53:50.388852 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:53:50.390703 (+ 1851us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38589: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":100,"threads_started":1}
W20260929 17:53:50.390966 9603 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:53:50.385592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47609 (local address 127.0.98.62:38589)
0929 17:53:50.385721 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:50.385726 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:50.385866 (+ 140us) server_negotiation.cc:407] Connection header received
0929 17:53:50.385983 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:50.385985 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:50.386032 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:50.386088 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:53:50.386531 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:50.387152 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:50.387960 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:50.388205 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:50.388284 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:50.390854 (+ 2570us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:47609: BlockingRecv error: recv got EOF from 127.0.98.3:47609 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 29 17:53:51 dist-test-slave-53nh krb5kdc[8569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 17:53:51 dist-test-slave-53nh krb5kdc[8569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 17:53:51.018230 9606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:51.013387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38589 (local address 127.0.98.1:33875)
0929 17:53:51.013654 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:51.013668 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 17:53:51.013759 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:53:51.013972 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:53:51.013974 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:53:51.014153 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:53:51.014250 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:51.014255 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:51.014937 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:51.014939 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:51.015483 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:51.015490 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:51.015848 (+ 358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:51.015851 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:51.015906 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:51.016388 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:53:51.016399 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:53:51.018111 (+ 1712us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38589: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":84,"threads_started":1}
W20260929 17:53:51.018399 9607 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:53:51.013452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33875 (local address 127.0.98.62:38589)
0929 17:53:51.013677 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:51.013681 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:51.013731 (+ 50us) server_negotiation.cc:407] Connection header received
0929 17:53:51.013828 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:51.013830 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:51.013872 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:51.013948 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:53:51.014341 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:51.014860 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:51.015561 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:51.015773 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:51.015836 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:51.018271 (+ 2435us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:33875: BlockingRecv error: recv got EOF from 127.0.98.1:33875 (error 108)
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":66,"threads_started":1}
Sep 29 17:53:51 dist-test-slave-53nh krb5kdc[8569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 17:53:51 dist-test-slave-53nh krb5kdc[8569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 17:53:51.215063 9608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:51.209986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38589 (local address 127.0.98.2:60591)
0929 17:53:51.210298 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:51.210310 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 17:53:51.210416 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:53:51.210660 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:53:51.210663 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:53:51.210856 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:53:51.210945 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:51.210951 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:51.211677 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:51.211681 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:51.212259 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:51.212267 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:51.212678 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:51.212682 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:51.212758 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:51.213248 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:53:51.213258 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:53:51.214921 (+ 1663us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38589: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":102,"threads_started":1}
W20260929 17:53:51.215205 9607 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:53:51.210086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60591 (local address 127.0.98.62:38589)
0929 17:53:51.210241 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:51.210245 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:51.210382 (+ 137us) server_negotiation.cc:407] Connection header received
0929 17:53:51.210504 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:51.210507 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:51.210551 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:51.210629 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:53:51.211064 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:51.211581 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:51.212371 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:51.212584 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:51.212667 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:51.215064 (+ 2397us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:60591: BlockingRecv error: recv got EOF from 127.0.98.2:60591 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 29 17:53:51 dist-test-slave-53nh krb5kdc[8569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 17:53:51 dist-test-slave-53nh krb5kdc[8569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 17:53:51.396713 9609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:51.391597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38589 (local address 127.0.98.3:46011)
0929 17:53:51.391860 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:51.391873 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 17:53:51.391963 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:53:51.392170 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:53:51.392172 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:53:51.392338 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:53:51.392429 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:51.392433 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:51.393132 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:51.393135 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:51.393695 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:51.393704 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:51.394093 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:51.394096 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:51.394159 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:51.394662 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:53:51.394674 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:53:51.396561 (+ 1887us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38589: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":92,"threads_started":1}
W20260929 17:53:51.396835 9607 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:53:51.391741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46011 (local address 127.0.98.62:38589)
0929 17:53:51.391873 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:51.391877 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:51.391920 (+ 43us) server_negotiation.cc:407] Connection header received
0929 17:53:51.392031 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:51.392034 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:51.392076 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:51.392138 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:53:51.392507 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:51.393056 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:51.393789 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:51.394002 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:51.394074 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:51.396734 (+ 2660us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:46011: BlockingRecv error: recv got EOF from 127.0.98.3:46011 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 29 17:53:52 dist-test-slave-53nh krb5kdc[8569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 17:53:52 dist-test-slave-53nh krb5kdc[8569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 17:53:52.025451 9610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:52.018947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38589 (local address 127.0.98.1:55135)
0929 17:53:52.019222 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:52.019235 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 17:53:52.019333 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:53:52.019650 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:53:52.019653 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:53:52.019841 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:53:52.019956 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:52.019965 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:52.020893 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:52.020897 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:52.021833 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:52.021845 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:52.022423 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:52.022427 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:52.022511 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:52.023049 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:53:52.023064 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:53:52.025289 (+ 2225us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38589: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":77,"threads_started":1}
W20260929 17:53:52.025692 9611 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:53:52.019005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55135 (local address 127.0.98.62:38589)
0929 17:53:52.019251 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:52.019256 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:52.019276 (+ 20us) server_negotiation.cc:407] Connection header received
0929 17:53:52.019450 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:52.019452 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:52.019515 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:52.019623 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:53:52.020154 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:52.020793 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:52.021946 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:52.022321 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:52.022412 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:52.025427 (+ 3015us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:55135: BlockingRecv error: recv got EOF from 127.0.98.1:55135 (error 108)
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":66,"threads_started":1}
Sep 29 17:53:52 dist-test-slave-53nh krb5kdc[8569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 17:53:52 dist-test-slave-53nh krb5kdc[8569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 17:53:52.221191 9612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:52.215894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38589 (local address 127.0.98.2:49593)
0929 17:53:52.216237 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:52.216250 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 17:53:52.216343 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:53:52.216623 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:53:52.216626 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:53:52.216847 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:53:52.216960 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:52.216966 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:52.217722 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:52.217726 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:52.218298 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:52.218306 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:52.218719 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:52.218722 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:52.218801 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:52.219295 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:53:52.219311 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:53:52.221052 (+ 1741us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38589: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":81,"threads_started":1}
W20260929 17:53:52.221302 9611 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:53:52.215958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49593 (local address 127.0.98.62:38589)
0929 17:53:52.216090 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:52.216094 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:52.216319 (+ 225us) server_negotiation.cc:407] Connection header received
0929 17:53:52.216444 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:52.216447 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:52.216499 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:52.216574 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:53:52.217096 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:52.217618 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:52.218402 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:52.218623 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:52.218715 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:52.221191 (+ 2476us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:49593: BlockingRecv error: recv got EOF from 127.0.98.2:49593 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 29 17:53:52 dist-test-slave-53nh krb5kdc[8569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 17:53:52 dist-test-slave-53nh krb5kdc[8569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 17:53:52.402719 9613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:52.397608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38589 (local address 127.0.98.3:42025)
0929 17:53:52.397866 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:52.397880 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 17:53:52.397978 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:53:52.398226 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:53:52.398228 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:53:52.398396 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:53:52.398485 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:52.398489 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:52.399198 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:52.399200 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:52.399762 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:52.399782 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:52.400213 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:52.400215 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:52.400275 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:52.400728 (+ 453us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:53:52.400739 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:53:52.402558 (+ 1819us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38589: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":75,"threads_started":1}
W20260929 17:53:52.402779 9611 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:53:52.397666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42025 (local address 127.0.98.62:38589)
0929 17:53:52.397806 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:52.397810 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:52.397942 (+ 132us) server_negotiation.cc:407] Connection header received
0929 17:53:52.398052 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:52.398054 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:52.398132 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:52.398200 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:53:52.398565 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:52.399115 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:52.399893 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:52.400118 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:52.400207 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:52.402679 (+ 2472us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:42025: BlockingRecv error: recv got EOF from 127.0.98.3:42025 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 29 17:53:53 dist-test-slave-53nh krb5kdc[8569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 17:53:53 dist-test-slave-53nh krb5kdc[8569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 17:53:53.031610 9614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:53.026302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38589 (local address 127.0.98.1:55095)
0929 17:53:53.026551 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:53.026567 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 17:53:53.026700 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:53:53.026965 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:53:53.026967 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:53:53.027165 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:53:53.027292 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:53.027299 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:53.028103 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:53.028106 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:53.028699 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:53.028709 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:53.029245 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:53.029248 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:53.029320 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:53.029814 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:53:53.029828 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:53:53.031487 (+ 1659us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38589: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":71,"threads_started":1}
W20260929 17:53:53.031769 9615 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:53:53.026365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55095 (local address 127.0.98.62:38589)
0929 17:53:53.026592 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:53.026596 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:53.026640 (+ 44us) server_negotiation.cc:407] Connection header received
0929 17:53:53.026788 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:53.026791 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:53.026844 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:53.026941 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:53:53.027400 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:53.027982 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:53.028816 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:53.029148 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:53.029233 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:53.031640 (+ 2407us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:55095: BlockingRecv error: recv got EOF from 127.0.98.1:55095 (error 108)
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":65,"threads_started":1}
Sep 29 17:53:53 dist-test-slave-53nh krb5kdc[8569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 17:53:53 dist-test-slave-53nh krb5kdc[8569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 17:53:53.227726 9616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:53.221996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38589 (local address 127.0.98.2:53831)
0929 17:53:53.222347 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:53.222363 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 17:53:53.222483 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:53:53.222794 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:53:53.222797 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:53:53.223090 (+ 293us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:53:53.223197 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:53.223203 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:53.224162 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:53.224165 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:53.224732 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:53.224739 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:53.225237 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:53.225240 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:53.225311 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:53.225827 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:53:53.225841 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:53:53.227612 (+ 1771us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38589: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":149,"threads_started":1}
W20260929 17:53:53.227960 9615 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:53:53.222177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53831 (local address 127.0.98.62:38589)
0929 17:53:53.222347 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:53.222352 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:53.222441 (+ 89us) server_negotiation.cc:407] Connection header received
0929 17:53:53.222614 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:53.222617 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:53.222671 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:53.222762 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:53:53.223317 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:53.224004 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:53.224863 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:53.225125 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:53.225209 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:53.227759 (+ 2550us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:53831: BlockingRecv error: recv got EOF from 127.0.98.2:53831 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Sep 29 17:53:53 dist-test-slave-53nh krb5kdc[8569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 17:53:53 dist-test-slave-53nh krb5kdc[8569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 17:53:53.409400 9617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:53.403449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38589 (local address 127.0.98.3:52679)
0929 17:53:53.403775 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:53.403789 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 17:53:53.403924 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:53:53.404261 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:53:53.404264 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:53:53.404470 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:53:53.404567 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:53.404574 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:53.405388 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:53.405392 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:53.405957 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:53.405966 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:53.406450 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:53.406454 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:53.406543 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:53.407103 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:53:53.407117 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:53:53.409237 (+ 2120us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38589: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":99,"threads_started":1}
W20260929 17:53:53.409533 9615 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:53:53.403543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52679 (local address 127.0.98.62:38589)
0929 17:53:53.403720 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:53.403725 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:53.403866 (+ 141us) server_negotiation.cc:407] Connection header received
0929 17:53:53.404086 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:53.404089 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:53.404148 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:53.404228 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:53:53.404686 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:53.405282 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:53.406085 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:53.406317 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:53.406411 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:53.409427 (+ 3016us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:52679: BlockingRecv error: recv got EOF from 127.0.98.3:52679 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Sep 29 17:53:54 dist-test-slave-53nh krb5kdc[8569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 17:53:54 dist-test-slave-53nh krb5kdc[8569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 17:53:54.038090 9618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:54.032457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38589 (local address 127.0.98.1:50155)
0929 17:53:54.032739 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:54.032753 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 17:53:54.032844 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:53:54.033154 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:53:54.033157 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:53:54.033352 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:53:54.033470 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:54.033478 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:54.034418 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:54.034422 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:54.035015 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:54.035025 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:54.035452 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:54.035455 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:54.035519 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:54.036032 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:53:54.036046 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:53:54.037944 (+ 1898us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38589: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":87,"threads_started":1}
W20260929 17:53:54.038205 9619 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:53:54.032622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50155 (local address 127.0.98.62:38589)
0929 17:53:54.032864 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:54.032867 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:54.032881 (+ 14us) server_negotiation.cc:407] Connection header received
0929 17:53:54.032922 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:54.032924 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:54.032965 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:54.033065 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:53:54.033613 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:54.034295 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:54.035122 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:54.035346 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:54.035420 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:54.038080 (+ 2660us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:50155: BlockingRecv error: recv got EOF from 127.0.98.1:50155 (error 108)
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":84,"threads_started":1}
Sep 29 17:53:54 dist-test-slave-53nh krb5kdc[8569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 17:53:54 dist-test-slave-53nh krb5kdc[8569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 17:53:54.234222 9620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:54.228600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38589 (local address 127.0.98.2:35745)
0929 17:53:54.228856 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:54.228869 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 17:53:54.228962 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:53:54.229282 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:53:54.229286 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:53:54.229457 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:53:54.229573 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:54.229579 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:54.230375 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:54.230379 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:54.230952 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:54.230960 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:54.231371 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:54.231374 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:54.231438 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:54.231955 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:53:54.231967 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:53:54.234074 (+ 2107us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38589: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":82,"threads_started":1}
W20260929 17:53:54.234303 9619 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:53:54.228788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35745 (local address 127.0.98.62:38589)
0929 17:53:54.228942 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:54.228945 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:54.228961 (+ 16us) server_negotiation.cc:407] Connection header received
0929 17:53:54.229085 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:54.229088 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:54.229139 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:54.229218 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:53:54.229697 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:54.230247 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:54.231054 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:54.231273 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:54.231347 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:54.234207 (+ 2860us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:35745: BlockingRecv error: recv got EOF from 127.0.98.2:35745 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Sep 29 17:53:54 dist-test-slave-53nh krb5kdc[8569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 17:53:54 dist-test-slave-53nh krb5kdc[8569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 17:53:54.416273 9621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:54.410358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38589 (local address 127.0.98.3:34957)
0929 17:53:54.410668 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:54.410686 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 17:53:54.410807 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:53:54.411071 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:53:54.411074 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:53:54.411320 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:53:54.411430 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:54.411437 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:54.412270 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:54.412273 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:54.412905 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:54.412915 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:54.413408 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:54.413411 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:54.413497 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:54.414105 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:53:54.414118 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:53:54.416122 (+ 2004us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38589: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":111,"threads_started":1}
W20260929 17:53:54.416385 9619 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:53:54.410414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34957 (local address 127.0.98.62:38589)
0929 17:53:54.410552 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:54.410557 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:54.410776 (+ 219us) server_negotiation.cc:407] Connection header received
0929 17:53:54.410900 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:54.410903 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:54.410953 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:54.411019 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:53:54.411607 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:54.412148 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:54.413050 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:54.413297 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:54.413380 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:54.416285 (+ 2905us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:34957: BlockingRecv error: recv got EOF from 127.0.98.3:34957 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Sep 29 17:53:55 dist-test-slave-53nh krb5kdc[8569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 17:53:55 dist-test-slave-53nh krb5kdc[8569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 17:53:55.048976 9622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:55.038798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38589 (local address 127.0.98.1:53303)
0929 17:53:55.039045 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:55.039058 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 17:53:55.039150 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:53:55.039354 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:53:55.039356 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:53:55.039524 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:53:55.039632 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:55.039637 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:55.040416 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:55.040419 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:55.041050 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:55.041071 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:55.041568 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:55.041571 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:55.041642 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:55.046877 (+ 5235us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:53:55.046889 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:53:55.048807 (+ 1918us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38589: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":89,"threads_started":1}
W20260929 17:53:55.049248 9623 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:53:55.038863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53303 (local address 127.0.98.62:38589)
0929 17:53:55.039129 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:55.039132 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:55.039149 (+ 17us) server_negotiation.cc:407] Connection header received
0929 17:53:55.039221 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:55.039223 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:55.039263 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:55.039335 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:53:55.039750 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:55.040320 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:55.041200 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:55.041463 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:55.041540 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:55.048949 (+ 7409us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:53303: BlockingRecv error: recv got EOF from 127.0.98.1:53303 (error 108)
Metrics: {"server-negotiator.queue_time_us":202,"thread_start_us":56,"threads_started":1}
Sep 29 17:53:55 dist-test-slave-53nh krb5kdc[8569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 17:53:55 dist-test-slave-53nh krb5kdc[8569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 17:53:55.241559 9624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:55.235157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38589 (local address 127.0.98.2:51009)
0929 17:53:55.235469 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:55.235481 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 17:53:55.235582 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:53:55.235882 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:53:55.235885 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:53:55.236127 (+ 242us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:53:55.236246 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:55.236254 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:55.237492 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:55.237495 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:55.238066 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:55.238075 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:55.238732 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:55.238735 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:55.238803 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:55.239325 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:53:55.239340 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:53:55.241420 (+ 2080us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38589: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":81,"threads_started":1}
W20260929 17:53:55.241745 9623 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:53:55.235278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51009 (local address 127.0.98.62:38589)
0929 17:53:55.235457 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:55.235462 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:55.235541 (+ 79us) server_negotiation.cc:407] Connection header received
0929 17:53:55.235678 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:55.235682 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:55.235740 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:55.235843 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:53:55.236443 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:55.237342 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:55.238232 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:55.238608 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:55.238719 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:55.241616 (+ 2897us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:51009: BlockingRecv error: recv got EOF from 127.0.98.2:51009 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 29 17:53:55 dist-test-slave-53nh krb5kdc[8569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 17:53:55 dist-test-slave-53nh krb5kdc[8569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 17:53:55.422960 9625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:55.417215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38589 (local address 127.0.98.3:58439)
0929 17:53:55.417485 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:55.417499 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 17:53:55.417633 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:53:55.417971 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:53:55.417973 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:53:55.418187 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:53:55.418284 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:55.418290 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:55.419144 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:55.419147 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:55.419732 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:55.419740 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:55.420196 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:55.420199 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:55.420256 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:55.420764 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:53:55.420775 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:53:55.422782 (+ 2007us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38589: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":102,"threads_started":1}
W20260929 17:53:55.423074 9623 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:53:55.417295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58439 (local address 127.0.98.62:38589)
0929 17:53:55.417507 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:55.417510 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:55.417581 (+ 71us) server_negotiation.cc:407] Connection header received
0929 17:53:55.417769 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:55.417772 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:55.417822 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:55.417899 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:53:55.418438 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:55.419005 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:55.419842 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:55.420083 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:55.420179 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:55.422954 (+ 2775us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:58439: BlockingRecv error: recv got EOF from 127.0.98.3:58439 (error 108)
Metrics: {"server-negotiator.queue_time_us":92}
Sep 29 17:53:56 dist-test-slave-53nh krb5kdc[8569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 17:53:56 dist-test-slave-53nh krb5kdc[8569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 17:53:56.055557 9626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:56.049949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38589 (local address 127.0.98.1:45179)
0929 17:53:56.050227 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:56.050240 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 17:53:56.050320 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:53:56.050542 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:53:56.050544 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:53:56.050705 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:53:56.050799 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:56.050804 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:56.051569 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:56.051573 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:56.052150 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:56.052158 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:56.052636 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:56.052639 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:56.052707 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:56.053483 (+ 776us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:53:56.053498 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:53:56.055434 (+ 1936us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38589: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":109,"threads_started":1}
W20260929 17:53:56.055742 9627 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:53:56.050043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45179 (local address 127.0.98.62:38589)
0929 17:53:56.050274 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:56.050277 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:56.050290 (+ 13us) server_negotiation.cc:407] Connection header received
0929 17:53:56.050397 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:56.050400 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:56.050442 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:56.050525 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:53:56.050880 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:56.051456 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:56.052263 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:56.052483 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:56.052574 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:56.055602 (+ 3028us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:45179: BlockingRecv error: recv got EOF from 127.0.98.1:45179 (error 108)
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":69,"threads_started":1}
Sep 29 17:53:56 dist-test-slave-53nh krb5kdc[8569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 17:53:56 dist-test-slave-53nh krb5kdc[8569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 17:53:56.248016 9628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:56.242624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38589 (local address 127.0.98.2:38159)
0929 17:53:56.242862 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:56.242875 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 17:53:56.242965 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:53:56.243181 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:53:56.243183 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:53:56.243347 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:53:56.243458 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:56.243464 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:56.244223 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:56.244226 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:56.244845 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:56.244855 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:56.245349 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:56.245353 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:56.245432 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:56.245933 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:53:56.245946 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:53:56.247880 (+ 1934us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38589: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":67,"threads_started":1}
W20260929 17:53:56.248140 9627 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:53:56.242717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38159 (local address 127.0.98.62:38589)
0929 17:53:56.242838 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:56.242841 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:56.242936 (+ 95us) server_negotiation.cc:407] Connection header received
0929 17:53:56.243046 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:56.243049 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:56.243097 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:56.243158 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:53:56.243552 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:56.244093 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:56.244969 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:56.245226 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:56.245307 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:56.248011 (+ 2704us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:38159: BlockingRecv error: recv got EOF from 127.0.98.2:38159 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
I20260929 17:53:56.331676 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskup3Soe/build/debug/bin/kudu with pid 9152
I20260929 17:53:56.336464 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskup3Soe/build/debug/bin/kudu with pid 9304
I20260929 17:53:56.341122 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskup3Soe/build/debug/bin/kudu with pid 9436
I20260929 17:53:56.345245 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskup3Soe/build/debug/bin/kudu with pid 9082
2026-09-29T17:53:56Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17844 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-29T17:53:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T17:53:56Z Disabled control of system clock
I20260929 17:53:56.372644 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskup3Soe/build/debug/bin/kudu
/tmp/dist-test-taskup3Soe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790704375975685-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790704375975685-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790704375975685-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790704375975685-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36267
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:35263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36267
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260929 17:53:56.479595 9632 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:53:56.479856 9632 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:53:56.479889 9632 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:53:56.483822 9632 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 17:53:56.483887 9632 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:53:56.483901 9632 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:53:56.483916 9632 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 17:53:56.483929 9632 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 17:53:56.488688 9632 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790704375975685-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790704375975685-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36267
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36267
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790704375975685-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9632
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790704375975685-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:53:56.489748 9632 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:53:56.491068 9632 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:53:56.499511 9637 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:53:56.499522 9640 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:53:56.499676 9638 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:53:56.499775 9632 server_base.cc:1034] running on GCE node
I20260929 17:53:56.500365 9632 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:53:56.501485 9632 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:53:56.502688 9632 hybrid_clock.cc:648] HybridClock initialized: now 1790704436502675 us; error 29 us; skew 500 ppm
I20260929 17:53:56.504645 9632 webserver.cc:460] Webserver started at http://127.0.98.62:35977/ using document root <none> and password file <none>
I20260929 17:53:56.505271 9632 fs_manager.cc:359] Metadata directory not provided
I20260929 17:53:56.505342 9632 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:53:56.505548 9632 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:53:56.507575 9632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790704375975685-392-0/minicluster-data/master-0/data/instance:
uuid: "ff3bae2d7a154d8f913ceb0e7c698b8a"
format_stamp: "Formatted at 2026-09-29 17:53:56 on dist-test-slave-53nh"
server_key: "d648b8671da17eac56fc0313ec178c6a"
server_key_iv: "493c89979361a768622000d6d4f10bf1"
server_key_version: "encryptionkey@0"
I20260929 17:53:56.508108 9632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790704375975685-392-0/minicluster-data/master-0/wal/instance:
uuid: "ff3bae2d7a154d8f913ceb0e7c698b8a"
format_stamp: "Formatted at 2026-09-29 17:53:56 on dist-test-slave-53nh"
server_key: "d648b8671da17eac56fc0313ec178c6a"
server_key_iv: "493c89979361a768622000d6d4f10bf1"
server_key_version: "encryptionkey@0"
I20260929 17:53:56.511750 9632 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260929 17:53:56.514286 9646 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:53:56.515579 9632 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 17:53:56.515694 9632 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790704375975685-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790704375975685-392-0/minicluster-data/master-0/wal
uuid: "ff3bae2d7a154d8f913ceb0e7c698b8a"
format_stamp: "Formatted at 2026-09-29 17:53:56 on dist-test-slave-53nh"
server_key: "d648b8671da17eac56fc0313ec178c6a"
server_key_iv: "493c89979361a768622000d6d4f10bf1"
server_key_version: "encryptionkey@0"
I20260929 17:53:56.515791 9632 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790704375975685-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790704375975685-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790704375975685-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:53:56.544721 9632 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:53:56.545495 9632 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:53:56.545668 9632 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:53:56.552354 9632 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36267
I20260929 17:53:56.552378 9698 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36267 every 8 connection(s)
I20260929 17:53:56.553320 9632 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790704375975685-392-0/minicluster-data/master-0/data/info.pb
I20260929 17:53:56.556281 9699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:53:56.558069 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskup3Soe/build/debug/bin/kudu as pid 9632
I20260929 17:53:56.558156 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790704375975685-392-0/minicluster-data/master-0/wal/instance
I20260929 17:53:56.558437 392 external_mini_cluster.cc:1428] Setting key fc62924d378b54867cd62939c63da640
I20260929 17:53:56.562510 9699 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff3bae2d7a154d8f913ceb0e7c698b8a: Bootstrap starting.
I20260929 17:53:56.564585 9699 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ff3bae2d7a154d8f913ceb0e7c698b8a: Neither blocks nor log segments found. Creating new log.
I20260929 17:53:56.565390 9699 log.cc:824] T 00000000000000000000000000000000 P ff3bae2d7a154d8f913ceb0e7c698b8a: Log is configured to *not* fsync() on all Append() calls
I20260929 17:53:56.567080 9702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:56.559790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38082 (local address 127.0.98.62:36267)
0929 17:53:56.560382 (+ 592us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:56.560396 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:56.560431 (+ 35us) server_negotiation.cc:407] Connection header received
0929 17:53:56.561105 (+ 674us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:56.561122 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:56.561492 (+ 370us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:56.561832 (+ 340us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:53:56.562233 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:56.563192 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:56.564000 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:56.564485 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:56.564632 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:56.565437 (+ 805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:53:56.565443 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:53:56.565459 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 17:53:56.565873 (+ 414us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:53:56.565926 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 17:53:56.565946 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:53:56.566029 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0929 17:53:56.566167 (+ 138us) server_negotiation.cc:299] Negotiation successful
0929 17:53:56.566410 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":258,"thread_start_us":119,"threads_started":1}
I20260929 17:53:56.567174 9699 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff3bae2d7a154d8f913ceb0e7c698b8a: No bootstrap required, opened a new log
I20260929 17:53:56.570537 9699 raft_consensus.cc:348] T 00000000000000000000000000000000 P ff3bae2d7a154d8f913ceb0e7c698b8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff3bae2d7a154d8f913ceb0e7c698b8a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36267 } }
I20260929 17:53:56.570745 9699 raft_consensus.cc:374] T 00000000000000000000000000000000 P ff3bae2d7a154d8f913ceb0e7c698b8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:53:56.570801 9699 raft_consensus.cc:729] T 00000000000000000000000000000000 P ff3bae2d7a154d8f913ceb0e7c698b8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff3bae2d7a154d8f913ceb0e7c698b8a, State: Initialized, Role: FOLLOWER
I20260929 17:53:56.571177 9699 consensus_queue.cc:260] T 00000000000000000000000000000000 P ff3bae2d7a154d8f913ceb0e7c698b8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff3bae2d7a154d8f913ceb0e7c698b8a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36267 } }
I20260929 17:53:56.571298 9699 raft_consensus.cc:388] T 00000000000000000000000000000000 P ff3bae2d7a154d8f913ceb0e7c698b8a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 17:53:56.571359 9699 raft_consensus.cc:482] T 00000000000000000000000000000000 P ff3bae2d7a154d8f913ceb0e7c698b8a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 17:53:56.571452 9699 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ff3bae2d7a154d8f913ceb0e7c698b8a [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:53:56.572325 9699 raft_consensus.cc:504] T 00000000000000000000000000000000 P ff3bae2d7a154d8f913ceb0e7c698b8a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff3bae2d7a154d8f913ceb0e7c698b8a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36267 } }
I20260929 17:53:56.572625 9699 leader_election.cc:302] T 00000000000000000000000000000000 P ff3bae2d7a154d8f913ceb0e7c698b8a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ff3bae2d7a154d8f913ceb0e7c698b8a; no voters:
I20260929 17:53:56.572904 9699 leader_election.cc:288] T 00000000000000000000000000000000 P ff3bae2d7a154d8f913ceb0e7c698b8a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 17:53:56.573055 9704 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ff3bae2d7a154d8f913ceb0e7c698b8a [term 1 FOLLOWER]: Leader election won for term 1
I20260929 17:53:56.573273 9704 raft_consensus.cc:686] T 00000000000000000000000000000000 P ff3bae2d7a154d8f913ceb0e7c698b8a [term 1 LEADER]: Becoming Leader. State: Replica: ff3bae2d7a154d8f913ceb0e7c698b8a, State: Running, Role: LEADER
I20260929 17:53:56.573585 9704 consensus_queue.cc:237] T 00000000000000000000000000000000 P ff3bae2d7a154d8f913ceb0e7c698b8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff3bae2d7a154d8f913ceb0e7c698b8a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36267 } }
I20260929 17:53:56.574156 9699 sys_catalog.cc:564] T 00000000000000000000000000000000 P ff3bae2d7a154d8f913ceb0e7c698b8a [sys.catalog]: configured and running, proceeding with master startup.
I20260929 17:53:56.575093 9705 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff3bae2d7a154d8f913ceb0e7c698b8a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ff3bae2d7a154d8f913ceb0e7c698b8a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff3bae2d7a154d8f913ceb0e7c698b8a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36267 } } }
I20260929 17:53:56.575222 9705 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff3bae2d7a154d8f913ceb0e7c698b8a [sys.catalog]: This master's current role is: LEADER
I20260929 17:53:56.575482 9706 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff3bae2d7a154d8f913ceb0e7c698b8a [sys.catalog]: SysCatalogTable state changed. Reason: New leader ff3bae2d7a154d8f913ceb0e7c698b8a. Latest consensus state: current_term: 1 leader_uuid: "ff3bae2d7a154d8f913ceb0e7c698b8a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff3bae2d7a154d8f913ceb0e7c698b8a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36267 } } }
I20260929 17:53:56.575608 9706 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff3bae2d7a154d8f913ceb0e7c698b8a [sys.catalog]: This master's current role is: LEADER
I20260929 17:53:56.575676 9713 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 17:53:56.578429 9713 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 17:53:56.583868 9713 catalog_manager.cc:1347] Generated new cluster ID: ea7a9669d15a4a85b009464be6d35789
I20260929 17:53:56.583931 9713 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 17:53:56.599076 9713 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 17:53:56.600315 9713 catalog_manager.cc:1504] Loading token signing keys...
I20260929 17:53:56.607163 9713 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ff3bae2d7a154d8f913ceb0e7c698b8a: Generated new TSK 0
I20260929 17:53:56.607986 9713 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 17:53:56.610813 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskup3Soe/build/debug/bin/kudu
/tmp/dist-test-taskup3Soe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790704375975685-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790704375975685-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790704375975685-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790704375975685-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36267
--builtin_ntp_servers=127.0.98.20:35263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260929 17:53:56.719937 9723 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:53:56.720233 9723 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:53:56.720275 9723 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:53:56.724320 9723 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:53:56.724385 9723 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:53:56.724438 9723 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 17:53:56.729200 9723 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790704375975685-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790704375975685-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790704375975685-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36267
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9723
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790704375975685-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:53:56.730217 9723 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:53:56.731524 9723 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:53:56.740794 9728 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:53:56.740814 9729 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:53:56.740794 9731 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:53:56.740980 9723 server_base.cc:1034] running on GCE node
I20260929 17:53:56.741494 9723 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:53:56.742096 9723 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:53:56.743263 9723 hybrid_clock.cc:648] HybridClock initialized: now 1790704436743249 us; error 27 us; skew 500 ppm
I20260929 17:53:56.745029 9723 webserver.cc:460] Webserver started at http://127.0.98.1:37427/ using document root <none> and password file <none>
I20260929 17:53:56.745589 9723 fs_manager.cc:359] Metadata directory not provided
I20260929 17:53:56.745643 9723 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:53:56.745806 9723 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:53:56.747752 9723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790704375975685-392-0/minicluster-data/ts-0/data/instance:
uuid: "9a4b8b3c4d794eeb803df367c6275bfb"
format_stamp: "Formatted at 2026-09-29 17:53:56 on dist-test-slave-53nh"
server_key: "d90d030fce76d143f16c562838bf19ce"
server_key_iv: "03fb1a3c1eaeda3b8e15888ea35b98d3"
server_key_version: "encryptionkey@0"
I20260929 17:53:56.748231 9723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790704375975685-392-0/minicluster-data/ts-0/wal/instance:
uuid: "9a4b8b3c4d794eeb803df367c6275bfb"
format_stamp: "Formatted at 2026-09-29 17:53:56 on dist-test-slave-53nh"
server_key: "d90d030fce76d143f16c562838bf19ce"
server_key_iv: "03fb1a3c1eaeda3b8e15888ea35b98d3"
server_key_version: "encryptionkey@0"
I20260929 17:53:56.751652 9723 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260929 17:53:56.754192 9737 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:53:56.755455 9723 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 17:53:56.755563 9723 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790704375975685-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790704375975685-392-0/minicluster-data/ts-0/wal
uuid: "9a4b8b3c4d794eeb803df367c6275bfb"
format_stamp: "Formatted at 2026-09-29 17:53:56 on dist-test-slave-53nh"
server_key: "d90d030fce76d143f16c562838bf19ce"
server_key_iv: "03fb1a3c1eaeda3b8e15888ea35b98d3"
server_key_version: "encryptionkey@0"
I20260929 17:53:56.755681 9723 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790704375975685-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790704375975685-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790704375975685-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:53:56.784611 9723 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:53:56.785319 9723 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:53:56.785475 9723 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:53:56.786080 9723 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 17:53:56.787009 9723 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:53:56.787052 9723 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:53:56.787092 9723 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:53:56.787120 9723 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 17:53:56.795001 9723 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38773
I20260929 17:53:56.795058 9850 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38773 every 8 connection(s)
I20260929 17:53:56.795881 9723 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790704375975685-392-0/minicluster-data/ts-0/data/info.pb
I20260929 17:53:56.796804 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskup3Soe/build/debug/bin/kudu as pid 9723
I20260929 17:53:56.796901 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790704375975685-392-0/minicluster-data/ts-0/wal/instance
I20260929 17:53:56.797269 392 external_mini_cluster.cc:1428] Setting key f3272925e45cfb69db467c02129533e4
I20260929 17:53:56.798995 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskup3Soe/build/debug/bin/kudu
/tmp/dist-test-taskup3Soe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790704375975685-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790704375975685-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790704375975685-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790704375975685-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36267
--builtin_ntp_servers=127.0.98.20:35263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260929 17:53:56.806275 9702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:56.797860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35311 (local address 127.0.98.62:36267)
0929 17:53:56.797995 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:56.797999 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:56.799153 (+ 1154us) server_negotiation.cc:407] Connection header received
0929 17:53:56.800502 (+ 1349us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:56.800506 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:56.800563 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:56.800658 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:53:56.801824 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:56.802418 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:56.804135 (+ 1717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:56.804359 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:56.804433 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:56.805736 (+ 1303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:53:56.805739 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:53:56.805741 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 17:53:56.805753 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:53:56.805774 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 17:53:56.805782 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:53:56.805859 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0929 17:53:56.806101 (+ 242us) server_negotiation.cc:299] Negotiation successful
0929 17:53:56.806160 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 17:53:56.807390 9853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:56.798273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36267 (local address 127.0.98.1:35311)
0929 17:53:56.799009 (+ 736us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:56.799059 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0929 17:53:56.800311 (+ 1252us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:53:56.800848 (+ 537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:53:56.800856 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:53:56.800941 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:53:56.801665 (+ 724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:56.801687 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:56.802566 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:56.802570 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:56.803990 (+ 1420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:56.804003 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:56.804509 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:56.804514 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:56.804722 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:56.805415 (+ 693us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 17:53:56.805417 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:53:56.805473 (+ 56us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 17:53:56.805562 (+ 89us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:53:56.805876 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:53:56.805885 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:53:56.805891 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 17:53:56.806045 (+ 154us) client_negotiation.cc:240] Negotiation successful
0929 17:53:56.806349 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":138,"threads_started":1}
I20260929 17:53:56.808542 9851 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36267
I20260929 17:53:56.808801 9851 heartbeater.cc:464] Registering TS with master...
I20260929 17:53:56.809485 9851 heartbeater.cc:511] Master 127.0.98.62:36267 requested a full tablet report, sending...
I20260929 17:53:56.811285 9663 ts_manager.cc:194] Registered new tserver with Master: 9a4b8b3c4d794eeb803df367c6275bfb (127.0.98.1:38773)
I20260929 17:53:56.812414 9663 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:35311
W20260929 17:53:56.909543 9854 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:53:56.909806 9854 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:53:56.909842 9854 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:53:56.913785 9854 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:53:56.913846 9854 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:53:56.913899 9854 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 17:53:56.918728 9854 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790704375975685-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790704375975685-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790704375975685-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36267
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9854
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790704375975685-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:53:56.919821 9854 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:53:56.921245 9854 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:53:56.930480 9862 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:53:56.930536 9854 server_base.cc:1034] running on GCE node
W20260929 17:53:56.930482 9860 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:53:56.930508 9859 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:53:56.931193 9854 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:53:56.931809 9854 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:53:56.932984 9854 hybrid_clock.cc:648] HybridClock initialized: now 1790704436932940 us; error 58 us; skew 500 ppm
I20260929 17:53:56.934849 9854 webserver.cc:460] Webserver started at http://127.0.98.2:35447/ using document root <none> and password file <none>
I20260929 17:53:56.935390 9854 fs_manager.cc:359] Metadata directory not provided
I20260929 17:53:56.935454 9854 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:53:56.935655 9854 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:53:56.937924 9854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790704375975685-392-0/minicluster-data/ts-1/data/instance:
uuid: "1f565c5c124b4b4089846f6ed2843061"
format_stamp: "Formatted at 2026-09-29 17:53:56 on dist-test-slave-53nh"
server_key: "43c53599608dff6773e19558665639e0"
server_key_iv: "b44f9a793e8da4a62495283b3e6054be"
server_key_version: "encryptionkey@0"
I20260929 17:53:56.938498 9854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790704375975685-392-0/minicluster-data/ts-1/wal/instance:
uuid: "1f565c5c124b4b4089846f6ed2843061"
format_stamp: "Formatted at 2026-09-29 17:53:56 on dist-test-slave-53nh"
server_key: "43c53599608dff6773e19558665639e0"
server_key_iv: "b44f9a793e8da4a62495283b3e6054be"
server_key_version: "encryptionkey@0"
I20260929 17:53:56.942395 9854 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 17:53:56.945093 9868 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:53:56.946379 9854 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 17:53:56.946497 9854 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790704375975685-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790704375975685-392-0/minicluster-data/ts-1/wal
uuid: "1f565c5c124b4b4089846f6ed2843061"
format_stamp: "Formatted at 2026-09-29 17:53:56 on dist-test-slave-53nh"
server_key: "43c53599608dff6773e19558665639e0"
server_key_iv: "b44f9a793e8da4a62495283b3e6054be"
server_key_version: "encryptionkey@0"
I20260929 17:53:56.946605 9854 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790704375975685-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790704375975685-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790704375975685-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:53:56.960223 9854 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:53:56.960889 9854 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:53:56.961093 9854 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:53:56.961678 9854 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 17:53:56.962576 9854 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:53:56.962621 9854 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:53:56.962673 9854 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:53:56.962700 9854 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 17:53:56.970753 9854 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44465
I20260929 17:53:56.970777 9981 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44465 every 8 connection(s)
I20260929 17:53:56.971652 9854 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790704375975685-392-0/minicluster-data/ts-1/data/info.pb
I20260929 17:53:56.975174 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskup3Soe/build/debug/bin/kudu as pid 9854
I20260929 17:53:56.975267 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790704375975685-392-0/minicluster-data/ts-1/wal/instance
I20260929 17:53:56.975518 392 external_mini_cluster.cc:1428] Setting key 69ef1fb34aa7d54d59cbbf724c7c13ca
I20260929 17:53:56.977525 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskup3Soe/build/debug/bin/kudu
/tmp/dist-test-taskup3Soe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790704375975685-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790704375975685-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790704375975685-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790704375975685-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36267
--builtin_ntp_servers=127.0.98.20:35263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260929 17:53:56.981460 9702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:56.973423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34749 (local address 127.0.98.62:36267)
0929 17:53:56.973542 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:56.973547 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:56.974635 (+ 1088us) server_negotiation.cc:407] Connection header received
0929 17:53:56.976220 (+ 1585us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:56.976223 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:56.976275 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:56.976363 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:53:56.977332 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:56.978058 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:56.979555 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:56.979764 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:56.979843 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:56.980924 (+ 1081us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:53:56.980927 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:53:56.980929 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 17:53:56.980940 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:53:56.980959 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 17:53:56.980967 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:53:56.981119 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0929 17:53:56.981260 (+ 141us) server_negotiation.cc:299] Negotiation successful
0929 17:53:56.981323 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260929 17:53:56.982280 9984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:56.973858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36267 (local address 127.0.98.2:34749)
0929 17:53:56.974493 (+ 635us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:56.974532 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 17:53:56.975982 (+ 1450us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:53:56.976498 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:53:56.976505 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:53:56.976588 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:53:56.977197 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:56.977211 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:56.978205 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:56.978208 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:56.979410 (+ 1202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:56.979422 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:56.979895 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:56.979898 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:56.980048 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:56.980661 (+ 613us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 17:53:56.980662 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:53:56.980702 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 17:53:56.980771 (+ 69us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:53:56.981120 (+ 349us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:53:56.981129 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:53:56.981134 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 17:53:56.981244 (+ 110us) client_negotiation.cc:240] Negotiation successful
0929 17:53:56.981541 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":119,"threads_started":1}
I20260929 17:53:56.983299 9982 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36267
I20260929 17:53:56.983517 9982 heartbeater.cc:464] Registering TS with master...
I20260929 17:53:56.984004 9982 heartbeater.cc:511] Master 127.0.98.62:36267 requested a full tablet report, sending...
I20260929 17:53:56.984983 9663 ts_manager.cc:194] Registered new tserver with Master: 1f565c5c124b4b4089846f6ed2843061 (127.0.98.2:44465)
I20260929 17:53:56.985764 9663 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:34749
W20260929 17:53:57.088991 9985 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:53:57.089289 9985 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:53:57.089325 9985 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:53:57.093315 9985 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:53:57.093374 9985 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:53:57.093427 9985 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 17:53:57.098095 9985 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790704375975685-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790704375975685-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790704375975685-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36267
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9985
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790704375975685-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:53:57.099123 9985 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:53:57.100447 9985 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:53:57.109951 9990 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:53:57.110112 9985 server_base.cc:1034] running on GCE node
W20260929 17:53:57.109951 9991 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:53:57.110328 9993 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:53:57.110772 9985 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:53:57.111492 9985 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:53:57.112684 9985 hybrid_clock.cc:648] HybridClock initialized: now 1790704437112667 us; error 28 us; skew 500 ppm
I20260929 17:53:57.114646 9985 webserver.cc:460] Webserver started at http://127.0.98.3:35393/ using document root <none> and password file <none>
I20260929 17:53:57.115187 9985 fs_manager.cc:359] Metadata directory not provided
I20260929 17:53:57.115245 9985 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:53:57.115409 9985 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:53:57.117460 9985 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790704375975685-392-0/minicluster-data/ts-2/data/instance:
uuid: "377cff2d34944a309b4765d23da098d8"
format_stamp: "Formatted at 2026-09-29 17:53:57 on dist-test-slave-53nh"
server_key: "9530d824fae919335552ec57282f95e0"
server_key_iv: "87d67341dc5b76bdcf2f55f94b7fdaa5"
server_key_version: "encryptionkey@0"
I20260929 17:53:57.117969 9985 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790704375975685-392-0/minicluster-data/ts-2/wal/instance:
uuid: "377cff2d34944a309b4765d23da098d8"
format_stamp: "Formatted at 2026-09-29 17:53:57 on dist-test-slave-53nh"
server_key: "9530d824fae919335552ec57282f95e0"
server_key_iv: "87d67341dc5b76bdcf2f55f94b7fdaa5"
server_key_version: "encryptionkey@0"
I20260929 17:53:57.121694 9985 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 17:53:57.124177 9999 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:53:57.125454 9985 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 17:53:57.125562 9985 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790704375975685-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790704375975685-392-0/minicluster-data/ts-2/wal
uuid: "377cff2d34944a309b4765d23da098d8"
format_stamp: "Formatted at 2026-09-29 17:53:57 on dist-test-slave-53nh"
server_key: "9530d824fae919335552ec57282f95e0"
server_key_iv: "87d67341dc5b76bdcf2f55f94b7fdaa5"
server_key_version: "encryptionkey@0"
I20260929 17:53:57.125670 9985 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790704375975685-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790704375975685-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790704375975685-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:53:57.152685 9985 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:53:57.153512 9985 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:53:57.153719 9985 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:53:57.154300 9985 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 17:53:57.155223 9985 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:53:57.155268 9985 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:53:57.155305 9985 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:53:57.155325 9985 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 17:53:57.163841 9985 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44963
I20260929 17:53:57.163897 10112 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44963 every 8 connection(s)
I20260929 17:53:57.164736 9985 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790704375975685-392-0/minicluster-data/ts-2/data/info.pb
I20260929 17:53:57.173642 9702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:57.166615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47713 (local address 127.0.98.62:36267)
0929 17:53:57.166756 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:57.166760 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:57.167803 (+ 1043us) server_negotiation.cc:407] Connection header received
0929 17:53:57.169123 (+ 1320us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:57.169125 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:57.169169 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:57.169240 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:53:57.170104 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:57.170626 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:57.171800 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:57.172024 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:57.172105 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:57.173237 (+ 1132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:53:57.173240 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:53:57.173243 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 17:53:57.173257 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:53:57.173278 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 17:53:57.173287 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:53:57.173358 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0929 17:53:57.173459 (+ 101us) server_negotiation.cc:299] Negotiation successful
0929 17:53:57.173521 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 17:53:57.174116 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskup3Soe/build/debug/bin/kudu as pid 9985
I20260929 17:53:57.174214 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790704375975685-392-0/minicluster-data/ts-2/wal/instance
I20260929 17:53:57.174502 392 external_mini_cluster.cc:1428] Setting key bf1af20ed0c333197f78c67d0205bfca
I20260929 17:53:57.174662 10115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:57.166984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36267 (local address 127.0.98.3:47713)
0929 17:53:57.167654 (+ 670us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:57.167695 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 17:53:57.168907 (+ 1212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:53:57.169394 (+ 487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:53:57.169401 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:53:57.169485 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:53:57.169979 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:57.169990 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:57.170767 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:57.170771 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:57.171678 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:57.171687 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:57.172194 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:57.172197 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:57.172350 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:57.172958 (+ 608us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 17:53:57.172959 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:53:57.173019 (+ 60us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 17:53:57.173091 (+ 72us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:53:57.173357 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:53:57.173362 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:53:57.173366 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 17:53:57.173451 (+ 85us) client_negotiation.cc:240] Negotiation successful
0929 17:53:57.173682 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":116,"threads_started":1}
I20260929 17:53:57.175765 10113 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36267
I20260929 17:53:57.175992 10113 heartbeater.cc:464] Registering TS with master...
I20260929 17:53:57.176551 10113 heartbeater.cc:511] Master 127.0.98.62:36267 requested a full tablet report, sending...
I20260929 17:53:57.177482 9662 ts_manager.cc:194] Registered new tserver with Master: 377cff2d34944a309b4765d23da098d8 (127.0.98.3:44963)
I20260929 17:53:57.178090 9662 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:47713
I20260929 17:53:57.188339 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260929 17:53:57.190955 9702 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:53:57.190117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38094 (local address 127.0.98.62:36267)
0929 17:53:57.190237 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:57.190241 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:57.190442 (+ 201us) server_negotiation.cc:407] Connection header received
0929 17:53:57.190540 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:57.190542 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:57.190587 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:57.190645 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:53:57.190832 (+ 187us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38094: BlockingRecv error: recv got EOF from 127.0.0.1:38094 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
I20260929 17:53:57.191053 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskup3Soe/build/debug/bin/kudu with pid 9723
I20260929 17:53:57.195931 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskup3Soe/build/debug/bin/kudu with pid 9854
I20260929 17:53:57.200153 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskup3Soe/build/debug/bin/kudu with pid 9985
I20260929 17:53:57.204324 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskup3Soe/build/debug/bin/kudu with pid 9632
2026-09-29T17:53:57Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (857 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-29T17:53:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T17:53:57Z Disabled control of system clock
I20260929 17:53:57.230293 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskup3Soe/build/debug/bin/kudu
/tmp/dist-test-taskup3Soe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790704375975685-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790704375975685-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790704375975685-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790704375975685-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36641
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:39909
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36641
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260929 17:53:57.337335 10124 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:53:57.337589 10124 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:53:57.337626 10124 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:53:57.341535 10124 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 17:53:57.341599 10124 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:53:57.341616 10124 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:53:57.341634 10124 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 17:53:57.341651 10124 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 17:53:57.346333 10124 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39909
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790704375975685-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790704375975685-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36641
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36641
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790704375975685-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10124
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790704375975685-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:53:57.347324 10124 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:53:57.348611 10124 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:53:57.356766 10130 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:53:57.356776 10132 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:53:57.356774 10129 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:53:57.356812 10124 server_base.cc:1034] running on GCE node
I20260929 17:53:57.357594 10124 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:53:57.358655 10124 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:53:57.359825 10124 hybrid_clock.cc:648] HybridClock initialized: now 1790704437359798 us; error 38 us; skew 500 ppm
I20260929 17:53:57.361704 10124 webserver.cc:460] Webserver started at http://127.0.98.62:46587/ using document root <none> and password file <none>
I20260929 17:53:57.362252 10124 fs_manager.cc:359] Metadata directory not provided
I20260929 17:53:57.362324 10124 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:53:57.362541 10124 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:53:57.364598 10124 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790704375975685-392-0/minicluster-data/master-0/data/instance:
uuid: "1a936b8c780c44a181e4ba430d0847e5"
format_stamp: "Formatted at 2026-09-29 17:53:57 on dist-test-slave-53nh"
server_key: "dbd7638c17243ebd83fdaf08943621b7"
server_key_iv: "22d4610d8f25f869905024a362bd30fe"
server_key_version: "encryptionkey@0"
I20260929 17:53:57.365196 10124 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790704375975685-392-0/minicluster-data/master-0/wal/instance:
uuid: "1a936b8c780c44a181e4ba430d0847e5"
format_stamp: "Formatted at 2026-09-29 17:53:57 on dist-test-slave-53nh"
server_key: "dbd7638c17243ebd83fdaf08943621b7"
server_key_iv: "22d4610d8f25f869905024a362bd30fe"
server_key_version: "encryptionkey@0"
I20260929 17:53:57.368829 10124 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260929 17:53:57.371318 10138 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:53:57.372640 10124 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 17:53:57.372751 10124 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790704375975685-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790704375975685-392-0/minicluster-data/master-0/wal
uuid: "1a936b8c780c44a181e4ba430d0847e5"
format_stamp: "Formatted at 2026-09-29 17:53:57 on dist-test-slave-53nh"
server_key: "dbd7638c17243ebd83fdaf08943621b7"
server_key_iv: "22d4610d8f25f869905024a362bd30fe"
server_key_version: "encryptionkey@0"
I20260929 17:53:57.372849 10124 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790704375975685-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790704375975685-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790704375975685-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:53:57.377322 10124 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:53:57.377880 10124 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:53:57.378027 10124 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:53:57.384234 10124 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36641
I20260929 17:53:57.384251 10190 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36641 every 8 connection(s)
I20260929 17:53:57.385205 10124 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790704375975685-392-0/minicluster-data/master-0/data/info.pb
I20260929 17:53:57.388260 10191 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:53:57.393697 10191 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a936b8c780c44a181e4ba430d0847e5: Bootstrap starting.
I20260929 17:53:57.395459 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskup3Soe/build/debug/bin/kudu as pid 10124
I20260929 17:53:57.395567 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790704375975685-392-0/minicluster-data/master-0/wal/instance
I20260929 17:53:57.395826 392 external_mini_cluster.cc:1428] Setting key f1fd49a63d0e1497a9d78522be1c0b9d
I20260929 17:53:57.396297 10191 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1a936b8c780c44a181e4ba430d0847e5: Neither blocks nor log segments found. Creating new log.
I20260929 17:53:57.397184 10191 log.cc:824] T 00000000000000000000000000000000 P 1a936b8c780c44a181e4ba430d0847e5: Log is configured to *not* fsync() on all Append() calls
I20260929 17:53:57.399189 10191 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a936b8c780c44a181e4ba430d0847e5: No bootstrap required, opened a new log
I20260929 17:53:57.401229 10194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:57.396946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57512 (local address 127.0.98.62:36641)
0929 17:53:57.397611 (+ 665us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:57.397626 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:57.397659 (+ 33us) server_negotiation.cc:407] Connection header received
0929 17:53:57.398290 (+ 631us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:57.398307 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:57.398611 (+ 304us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:57.398890 (+ 279us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:53:57.399882 (+ 992us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:53:57.399887 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:53:57.399899 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 17:53:57.399909 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:53:57.399965 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 17:53:57.399982 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:53:57.400096 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0929 17:53:57.400202 (+ 106us) server_negotiation.cc:299] Negotiation successful
0929 17:53:57.400448 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":116,"threads_started":1}
I20260929 17:53:57.403606 10191 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1a936b8c780c44a181e4ba430d0847e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a936b8c780c44a181e4ba430d0847e5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36641 } }
I20260929 17:53:57.403846 10191 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1a936b8c780c44a181e4ba430d0847e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:53:57.403901 10191 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1a936b8c780c44a181e4ba430d0847e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a936b8c780c44a181e4ba430d0847e5, State: Initialized, Role: FOLLOWER
I20260929 17:53:57.404318 10191 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1a936b8c780c44a181e4ba430d0847e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a936b8c780c44a181e4ba430d0847e5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36641 } }
I20260929 17:53:57.404444 10191 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1a936b8c780c44a181e4ba430d0847e5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 17:53:57.404521 10191 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1a936b8c780c44a181e4ba430d0847e5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 17:53:57.404614 10191 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1a936b8c780c44a181e4ba430d0847e5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:53:57.405612 10191 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1a936b8c780c44a181e4ba430d0847e5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a936b8c780c44a181e4ba430d0847e5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36641 } }
I20260929 17:53:57.405942 10191 leader_election.cc:302] T 00000000000000000000000000000000 P 1a936b8c780c44a181e4ba430d0847e5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1a936b8c780c44a181e4ba430d0847e5; no voters:
I20260929 17:53:57.406239 10191 leader_election.cc:288] T 00000000000000000000000000000000 P 1a936b8c780c44a181e4ba430d0847e5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 17:53:57.406342 10196 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1a936b8c780c44a181e4ba430d0847e5 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 17:53:57.406529 10196 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1a936b8c780c44a181e4ba430d0847e5 [term 1 LEADER]: Becoming Leader. State: Replica: 1a936b8c780c44a181e4ba430d0847e5, State: Running, Role: LEADER
I20260929 17:53:57.406862 10196 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1a936b8c780c44a181e4ba430d0847e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a936b8c780c44a181e4ba430d0847e5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36641 } }
I20260929 17:53:57.407415 10191 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1a936b8c780c44a181e4ba430d0847e5 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 17:53:57.408485 10198 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a936b8c780c44a181e4ba430d0847e5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1a936b8c780c44a181e4ba430d0847e5. Latest consensus state: current_term: 1 leader_uuid: "1a936b8c780c44a181e4ba430d0847e5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a936b8c780c44a181e4ba430d0847e5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36641 } } }
I20260929 17:53:57.408485 10197 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a936b8c780c44a181e4ba430d0847e5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1a936b8c780c44a181e4ba430d0847e5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a936b8c780c44a181e4ba430d0847e5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36641 } } }
I20260929 17:53:57.408660 10198 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a936b8c780c44a181e4ba430d0847e5 [sys.catalog]: This master's current role is: LEADER
I20260929 17:53:57.408660 10197 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a936b8c780c44a181e4ba430d0847e5 [sys.catalog]: This master's current role is: LEADER
W20260929 17:53:57.412199 10211 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1a936b8c780c44a181e4ba430d0847e5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 17:53:57.412282 10211 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 17:53:57.412358 10205 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 17:53:57.413087 10205 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 17:53:57.419121 10205 catalog_manager.cc:1347] Generated new cluster ID: 3426807cf6024178ba8eadab509ddbaf
I20260929 17:53:57.419217 10205 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 17:53:57.440413 10205 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 17:53:57.440742 10205 catalog_manager.cc:1504] Loading token signing keys...
I20260929 17:53:57.453028 10205 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1a936b8c780c44a181e4ba430d0847e5: Generated new TSK 0
I20260929 17:53:57.453866 10205 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 17:53:57.463227 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskup3Soe/build/debug/bin/kudu
/tmp/dist-test-taskup3Soe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790704375975685-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790704375975685-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790704375975685-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790704375975685-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36641
--builtin_ntp_servers=127.0.98.20:39909
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260929 17:53:57.574651 10215 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:53:57.574937 10215 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:53:57.574977 10215 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:53:57.579185 10215 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:53:57.579246 10215 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:53:57.579305 10215 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 17:53:57.584299 10215 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39909
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790704375975685-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790704375975685-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790704375975685-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36641
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10215
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790704375975685-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:53:57.585479 10215 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:53:57.586875 10215 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:53:57.597079 10220 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:53:57.597085 10221 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:53:57.597254 10223 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:53:57.597481 10215 server_base.cc:1034] running on GCE node
I20260929 17:53:57.597929 10215 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:53:57.598639 10215 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:53:57.599825 10215 hybrid_clock.cc:648] HybridClock initialized: now 1790704437599802 us; error 33 us; skew 500 ppm
I20260929 17:53:57.601869 10215 webserver.cc:460] Webserver started at http://127.0.98.1:34741/ using document root <none> and password file <none>
I20260929 17:53:57.602474 10215 fs_manager.cc:359] Metadata directory not provided
I20260929 17:53:57.602546 10215 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:53:57.602780 10215 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:53:57.604919 10215 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790704375975685-392-0/minicluster-data/ts-0/data/instance:
uuid: "337229de13c743568bc95df32067fb7b"
format_stamp: "Formatted at 2026-09-29 17:53:57 on dist-test-slave-53nh"
server_key: "9e82a131b183bfded2bc7b42e509235e"
server_key_iv: "63d36c4a6f441623162f96255506b396"
server_key_version: "encryptionkey@0"
I20260929 17:53:57.605551 10215 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790704375975685-392-0/minicluster-data/ts-0/wal/instance:
uuid: "337229de13c743568bc95df32067fb7b"
format_stamp: "Formatted at 2026-09-29 17:53:57 on dist-test-slave-53nh"
server_key: "9e82a131b183bfded2bc7b42e509235e"
server_key_iv: "63d36c4a6f441623162f96255506b396"
server_key_version: "encryptionkey@0"
I20260929 17:53:57.609594 10215 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260929 17:53:57.612057 10229 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:53:57.613413 10215 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 17:53:57.613546 10215 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790704375975685-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790704375975685-392-0/minicluster-data/ts-0/wal
uuid: "337229de13c743568bc95df32067fb7b"
format_stamp: "Formatted at 2026-09-29 17:53:57 on dist-test-slave-53nh"
server_key: "9e82a131b183bfded2bc7b42e509235e"
server_key_iv: "63d36c4a6f441623162f96255506b396"
server_key_version: "encryptionkey@0"
I20260929 17:53:57.613649 10215 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790704375975685-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790704375975685-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790704375975685-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:53:57.618433 10215 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:53:57.619124 10215 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:53:57.619318 10215 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:53:57.619899 10215 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 17:53:57.620841 10215 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:53:57.620890 10215 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:53:57.620947 10215 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:53:57.620985 10215 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 17:53:57.630220 10215 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36927
I20260929 17:53:57.630249 10342 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36927 every 8 connection(s)
I20260929 17:53:57.631165 10215 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790704375975685-392-0/minicluster-data/ts-0/data/info.pb
I20260929 17:53:57.637308 10194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:57.632988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56333 (local address 127.0.98.62:36641)
0929 17:53:57.633150 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:57.633153 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:57.634280 (+ 1127us) server_negotiation.cc:407] Connection header received
0929 17:53:57.635590 (+ 1310us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:57.635594 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:57.635638 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:57.635713 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:53:57.636819 (+ 1106us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:53:57.636822 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:53:57.636824 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 17:53:57.636837 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:53:57.636865 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 17:53:57.636873 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:53:57.636957 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0929 17:53:57.637121 (+ 164us) server_negotiation.cc:299] Negotiation successful
0929 17:53:57.637170 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 17:53:57.638156 10345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:57.633404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36641 (local address 127.0.98.1:56333)
0929 17:53:57.634111 (+ 707us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:57.634154 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 17:53:57.635371 (+ 1217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:53:57.635898 (+ 527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:53:57.635905 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:53:57.636001 (+ 96us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:53:57.636548 (+ 547us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 17:53:57.636550 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:53:57.636591 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 17:53:57.636661 (+ 70us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:53:57.636945 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:53:57.636953 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:53:57.636958 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 17:53:57.637147 (+ 189us) client_negotiation.cc:240] Negotiation successful
0929 17:53:57.637389 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":113,"threads_started":1}
I20260929 17:53:57.639173 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskup3Soe/build/debug/bin/kudu as pid 10215
I20260929 17:53:57.639274 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790704375975685-392-0/minicluster-data/ts-0/wal/instance
I20260929 17:53:57.639412 10343 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36641
I20260929 17:53:57.639581 392 external_mini_cluster.cc:1428] Setting key b4a88b1b9ba995f4f8965168cf230974
I20260929 17:53:57.639667 10343 heartbeater.cc:464] Registering TS with master...
I20260929 17:53:57.640100 10343 heartbeater.cc:511] Master 127.0.98.62:36641 requested a full tablet report, sending...
I20260929 17:53:57.641505 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskup3Soe/build/debug/bin/kudu
/tmp/dist-test-taskup3Soe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790704375975685-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790704375975685-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790704375975685-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790704375975685-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36641
--builtin_ntp_servers=127.0.98.20:39909
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260929 17:53:57.641649 10155 ts_manager.cc:194] Registered new tserver with Master: 337229de13c743568bc95df32067fb7b (127.0.98.1:36927)
W20260929 17:53:57.748459 10346 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:53:57.748757 10346 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:53:57.748798 10346 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:53:57.752854 10346 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:53:57.752921 10346 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:53:57.752977 10346 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 17:53:57.757829 10346 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39909
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790704375975685-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790704375975685-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790704375975685-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36641
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10346
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790704375975685-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:53:57.759020 10346 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:53:57.760537 10346 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:53:57.769899 10351 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:53:57.769899 10354 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:53:57.770009 10346 server_base.cc:1034] running on GCE node
W20260929 17:53:57.769930 10352 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:53:57.770545 10346 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:53:57.771164 10346 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:53:57.772315 10346 hybrid_clock.cc:648] HybridClock initialized: now 1790704437772294 us; error 31 us; skew 500 ppm
I20260929 17:53:57.774101 10346 webserver.cc:460] Webserver started at http://127.0.98.2:39857/ using document root <none> and password file <none>
I20260929 17:53:57.774657 10346 fs_manager.cc:359] Metadata directory not provided
I20260929 17:53:57.774720 10346 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:53:57.774904 10346 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:53:57.777058 10346 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790704375975685-392-0/minicluster-data/ts-1/data/instance:
uuid: "ed8db33d75e14ad08c6756442c4fb9ea"
format_stamp: "Formatted at 2026-09-29 17:53:57 on dist-test-slave-53nh"
server_key: "9d1c705805835340a75f92db400866c4"
server_key_iv: "19894058c2d46fdcf4e1e247f2e2cbaf"
server_key_version: "encryptionkey@0"
I20260929 17:53:57.777632 10346 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790704375975685-392-0/minicluster-data/ts-1/wal/instance:
uuid: "ed8db33d75e14ad08c6756442c4fb9ea"
format_stamp: "Formatted at 2026-09-29 17:53:57 on dist-test-slave-53nh"
server_key: "9d1c705805835340a75f92db400866c4"
server_key_iv: "19894058c2d46fdcf4e1e247f2e2cbaf"
server_key_version: "encryptionkey@0"
I20260929 17:53:57.781272 10346 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260929 17:53:57.783851 10360 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:53:57.785197 10346 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 17:53:57.785310 10346 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790704375975685-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790704375975685-392-0/minicluster-data/ts-1/wal
uuid: "ed8db33d75e14ad08c6756442c4fb9ea"
format_stamp: "Formatted at 2026-09-29 17:53:57 on dist-test-slave-53nh"
server_key: "9d1c705805835340a75f92db400866c4"
server_key_iv: "19894058c2d46fdcf4e1e247f2e2cbaf"
server_key_version: "encryptionkey@0"
I20260929 17:53:57.785413 10346 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790704375975685-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790704375975685-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790704375975685-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:53:57.790023 10346 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:53:57.790658 10346 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:53:57.790819 10346 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:53:57.791368 10346 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 17:53:57.792285 10346 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:53:57.792332 10346 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:53:57.792382 10346 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:53:57.792411 10346 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 17:53:57.800621 10346 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38763
I20260929 17:53:57.800653 10473 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38763 every 8 connection(s)
I20260929 17:53:57.801568 10346 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790704375975685-392-0/minicluster-data/ts-1/data/info.pb
I20260929 17:53:57.807696 10194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:57.803500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43973 (local address 127.0.98.62:36641)
0929 17:53:57.803648 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:57.803652 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:57.804764 (+ 1112us) server_negotiation.cc:407] Connection header received
0929 17:53:57.806082 (+ 1318us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:57.806085 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:57.806124 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:57.806203 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:53:57.807286 (+ 1083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:53:57.807289 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:53:57.807291 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 17:53:57.807303 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:53:57.807322 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 17:53:57.807330 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:53:57.807405 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0929 17:53:57.807499 (+ 94us) server_negotiation.cc:299] Negotiation successful
0929 17:53:57.807536 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 17:53:57.807852 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskup3Soe/build/debug/bin/kudu as pid 10346
I20260929 17:53:57.807960 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790704375975685-392-0/minicluster-data/ts-1/wal/instance
I20260929 17:53:57.808264 392 external_mini_cluster.cc:1428] Setting key b7365a722fa9796a8d75b8f16a224cee
I20260929 17:53:57.808702 10476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:57.803935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36641 (local address 127.0.98.2:43973)
0929 17:53:57.804620 (+ 685us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:57.804662 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 17:53:57.805871 (+ 1209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:53:57.806352 (+ 481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:53:57.806359 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:53:57.806449 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:53:57.806982 (+ 533us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 17:53:57.806983 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:53:57.807025 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 17:53:57.807147 (+ 122us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:53:57.807394 (+ 247us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:53:57.807399 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:53:57.807403 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 17:53:57.807527 (+ 124us) client_negotiation.cc:240] Negotiation successful
0929 17:53:57.807751 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":115,"threads_started":1}
I20260929 17:53:57.809887 10474 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36641
I20260929 17:53:57.810134 10474 heartbeater.cc:464] Registering TS with master...
I20260929 17:53:57.810273 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskup3Soe/build/debug/bin/kudu
/tmp/dist-test-taskup3Soe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790704375975685-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790704375975685-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790704375975685-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790704375975685-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36641
--builtin_ntp_servers=127.0.98.20:39909
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260929 17:53:57.810551 10474 heartbeater.cc:511] Master 127.0.98.62:36641 requested a full tablet report, sending...
I20260929 17:53:57.811525 10154 ts_manager.cc:194] Registered new tserver with Master: ed8db33d75e14ad08c6756442c4fb9ea (127.0.98.2:38763)
W20260929 17:53:57.919180 10477 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:53:57.919435 10477 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:53:57.919488 10477 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:53:57.923491 10477 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:53:57.923545 10477 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:53:57.923623 10477 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 17:53:57.928342 10477 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39909
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790704375975685-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790704375975685-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790704375975685-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36641
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10477
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790704375975685-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:53:57.929389 10477 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:53:57.930687 10477 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:53:57.939709 10485 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:53:57.939874 10477 server_base.cc:1034] running on GCE node
W20260929 17:53:57.939716 10482 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:53:57.939709 10483 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:53:57.940379 10477 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:53:57.940991 10477 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:53:57.942206 10477 hybrid_clock.cc:648] HybridClock initialized: now 1790704437942181 us; error 37 us; skew 500 ppm
I20260929 17:53:57.943995 10477 webserver.cc:460] Webserver started at http://127.0.98.3:35057/ using document root <none> and password file <none>
I20260929 17:53:57.944550 10477 fs_manager.cc:359] Metadata directory not provided
I20260929 17:53:57.944610 10477 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:53:57.944793 10477 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:53:57.946909 10477 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790704375975685-392-0/minicluster-data/ts-2/data/instance:
uuid: "a37541cfeb2e4894afaf46cd799386a1"
format_stamp: "Formatted at 2026-09-29 17:53:57 on dist-test-slave-53nh"
server_key: "2aa4f99492788e6faa22e0abe300594c"
server_key_iv: "06c363884dc413ed5e179fd81250147a"
server_key_version: "encryptionkey@0"
I20260929 17:53:57.947438 10477 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790704375975685-392-0/minicluster-data/ts-2/wal/instance:
uuid: "a37541cfeb2e4894afaf46cd799386a1"
format_stamp: "Formatted at 2026-09-29 17:53:57 on dist-test-slave-53nh"
server_key: "2aa4f99492788e6faa22e0abe300594c"
server_key_iv: "06c363884dc413ed5e179fd81250147a"
server_key_version: "encryptionkey@0"
I20260929 17:53:57.951109 10477 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 17:53:57.953536 10491 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:53:57.954926 10477 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 17:53:57.955034 10477 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790704375975685-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790704375975685-392-0/minicluster-data/ts-2/wal
uuid: "a37541cfeb2e4894afaf46cd799386a1"
format_stamp: "Formatted at 2026-09-29 17:53:57 on dist-test-slave-53nh"
server_key: "2aa4f99492788e6faa22e0abe300594c"
server_key_iv: "06c363884dc413ed5e179fd81250147a"
server_key_version: "encryptionkey@0"
I20260929 17:53:57.955163 10477 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790704375975685-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790704375975685-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790704375975685-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:53:57.959685 10477 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:53:57.960260 10477 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:53:57.960408 10477 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:53:57.960919 10477 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 17:53:57.961812 10477 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:53:57.961858 10477 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:53:57.961903 10477 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:53:57.961930 10477 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 17:53:57.971189 10477 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33769
I20260929 17:53:57.971232 10604 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33769 every 8 connection(s)
I20260929 17:53:57.972111 10477 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790704375975685-392-0/minicluster-data/ts-2/data/info.pb
I20260929 17:53:57.976080 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskup3Soe/build/debug/bin/kudu as pid 10477
I20260929 17:53:57.976197 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790704375975685-392-0/minicluster-data/ts-2/wal/instance
I20260929 17:53:57.976517 392 external_mini_cluster.cc:1428] Setting key 008ed3beb852a4458008ca81c92a7366
I20260929 17:53:57.978361 10194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:57.973935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36923 (local address 127.0.98.62:36641)
0929 17:53:57.974118 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:57.974122 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:57.975177 (+ 1055us) server_negotiation.cc:407] Connection header received
0929 17:53:57.976675 (+ 1498us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:57.976678 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:57.976717 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:57.976794 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:53:57.977931 (+ 1137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:53:57.977935 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:53:57.977937 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 17:53:57.977947 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:53:57.977983 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 17:53:57.977993 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:53:57.978075 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0929 17:53:57.978193 (+ 118us) server_negotiation.cc:299] Negotiation successful
0929 17:53:57.978233 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260929 17:53:57.979190 10607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:57.974354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36641 (local address 127.0.98.3:36923)
0929 17:53:57.975017 (+ 663us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:57.975066 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 17:53:57.976407 (+ 1341us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:53:57.976978 (+ 571us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:53:57.976985 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:53:57.977113 (+ 128us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:53:57.977696 (+ 583us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 17:53:57.977698 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:53:57.977755 (+ 57us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 17:53:57.977829 (+ 74us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:53:57.978071 (+ 242us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:53:57.978076 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:53:57.978080 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 17:53:57.978211 (+ 131us) client_negotiation.cc:240] Negotiation successful
0929 17:53:57.978411 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":120,"threads_started":1}
I20260929 17:53:57.980203 10605 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36641
I20260929 17:53:57.980398 10605 heartbeater.cc:464] Registering TS with master...
I20260929 17:53:57.980758 10605 heartbeater.cc:511] Master 127.0.98.62:36641 requested a full tablet report, sending...
I20260929 17:53:57.981730 10154 ts_manager.cc:194] Registered new tserver with Master: a37541cfeb2e4894afaf46cd799386a1 (127.0.98.3:33769)
I20260929 17:53:57.990495 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260929 17:53:57.993170 10194 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:53:57.992363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57520 (local address 127.0.98.62:36641)
0929 17:53:57.992531 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:57.992535 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:57.992634 (+ 99us) server_negotiation.cc:407] Connection header received
0929 17:53:57.992727 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:57.992729 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:57.992761 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:57.992825 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:53:57.993037 (+ 212us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57520: BlockingRecv error: recv got EOF from 127.0.0.1:57520 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 17:53:57.993584 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskup3Soe/build/debug/bin/kudu with pid 10215
I20260929 17:53:57.998723 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskup3Soe/build/debug/bin/kudu with pid 10346
I20260929 17:53:58.003338 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskup3Soe/build/debug/bin/kudu with pid 10477
I20260929 17:53:58.008034 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskup3Soe/build/debug/bin/kudu with pid 10124
2026-09-29T17:53:58Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (804 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704375975685-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 17:53:58 dist-test-slave-53nh krb5kdc[10615](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 17:53:58 dist-test-slave-53nh krb5kdc[10615](info): set up 2 sockets
Sep 29 17:53:58 dist-test-slave-53nh krb5kdc[10615](info): commencing operation
krb5kdc: starting...
W20260929 17:53:58.961647 392 mini_kdc.cc:121] Time spent starting KDC: real 0.934s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704375975685-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704375975685-392-0/krb5kdc/test-user.keytab.
Sep 29 17:53:59 dist-test-slave-53nh krb5kdc[10615](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704439, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T17:53:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T17:53:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704375975685-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704375975685-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 17:53:59.077864 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskup3Soe/build/debug/bin/kudu
/tmp/dist-test-taskup3Soe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704375975685-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704375975685-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704375975685-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704375975685-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35235
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704375975685-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35235
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704375975685-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704375975685-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704375975685-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:53:59.183910 10628 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:53:59.184192 10628 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:53:59.184231 10628 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:53:59.188206 10628 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 17:53:59.188272 10628 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:53:59.188292 10628 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:53:59.188309 10628 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 17:53:59.188325 10628 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 17:53:59.193578 10628 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704375975685-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704375975685-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35235
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704375975685-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35235
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704375975685-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10628
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704375975685-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:53:59.194607 10628 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:53:59.195592 10628 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:53:59.203792 10633 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:53:59.203797 10634 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:53:59.203878 10628 server_base.cc:1034] running on GCE node
W20260929 17:53:59.203792 10636 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:53:59.204600 10628 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:53:59.205734 10628 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:53:59.206882 10628 hybrid_clock.cc:648] HybridClock initialized: now 1790704439206848 us; error 46 us; skew 500 ppm
Sep 29 17:53:59 dist-test-slave-53nh krb5kdc[10615](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704439, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:53:59.209586 10628 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 17:53:59.210501 10628 webserver.cc:460] Webserver started at http://127.0.98.62:39467/ using document root <none> and password file <none>
I20260929 17:53:59.211024 10628 fs_manager.cc:359] Metadata directory not provided
I20260929 17:53:59.211084 10628 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:53:59.211277 10628 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:53:59.213306 10628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704375975685-392-0/minicluster-data/master-0/data/instance:
uuid: "c1225753a4f646bb8ac4cdcf73becc3b"
format_stamp: "Formatted at 2026-09-29 17:53:59 on dist-test-slave-53nh"
server_key: "c8513a133a351608ee80c55b114dc83c"
server_key_iv: "dd7c4433a53ab9e03c139a70abf61df6"
server_key_version: "encryptionkey@0"
I20260929 17:53:59.213828 10628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704375975685-392-0/minicluster-data/master-0/wal/instance:
uuid: "c1225753a4f646bb8ac4cdcf73becc3b"
format_stamp: "Formatted at 2026-09-29 17:53:59 on dist-test-slave-53nh"
server_key: "c8513a133a351608ee80c55b114dc83c"
server_key_iv: "dd7c4433a53ab9e03c139a70abf61df6"
server_key_version: "encryptionkey@0"
I20260929 17:53:59.217422 10628 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260929 17:53:59.219717 10643 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:53:59.220916 10628 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 17:53:59.221048 10628 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704375975685-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704375975685-392-0/minicluster-data/master-0/wal
uuid: "c1225753a4f646bb8ac4cdcf73becc3b"
format_stamp: "Formatted at 2026-09-29 17:53:59 on dist-test-slave-53nh"
server_key: "c8513a133a351608ee80c55b114dc83c"
server_key_iv: "dd7c4433a53ab9e03c139a70abf61df6"
server_key_version: "encryptionkey@0"
I20260929 17:53:59.221143 10628 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704375975685-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704375975685-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704375975685-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:53:59.230863 10628 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:53:59.233973 10628 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:53:59.234153 10628 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:53:59.240921 10628 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35235
I20260929 17:53:59.241220 10695 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35235 every 8 connection(s)
I20260929 17:53:59.241915 10628 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704375975685-392-0/minicluster-data/master-0/data/info.pb
I20260929 17:53:59.243113 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskup3Soe/build/debug/bin/kudu as pid 10628
I20260929 17:53:59.243250 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704375975685-392-0/minicluster-data/master-0/wal/instance
I20260929 17:53:59.243541 392 external_mini_cluster.cc:1428] Setting key e27b1039101f3c22c4aaef713b67e216
I20260929 17:53:59.245082 10696 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:53:59.251223 10696 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c1225753a4f646bb8ac4cdcf73becc3b: Bootstrap starting.
Sep 29 17:53:59 dist-test-slave-53nh krb5kdc[10615](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704439, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 17:53:59.253804 10696 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c1225753a4f646bb8ac4cdcf73becc3b: Neither blocks nor log segments found. Creating new log.
I20260929 17:53:59.254590 10696 log.cc:824] T 00000000000000000000000000000000 P c1225753a4f646bb8ac4cdcf73becc3b: Log is configured to *not* fsync() on all Append() calls
I20260929 17:53:59.256654 10696 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c1225753a4f646bb8ac4cdcf73becc3b: No bootstrap required, opened a new log
I20260929 17:53:59.257912 10699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:59.245081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53772 (local address 127.0.98.62:35235)
0929 17:53:59.245473 (+ 392us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:59.245487 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:59.245534 (+ 47us) server_negotiation.cc:407] Connection header received
0929 17:53:59.246164 (+ 630us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:59.246180 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:59.246517 (+ 337us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:59.246801 (+ 284us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:53:59.247646 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:59.248649 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:59.249791 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:59.250287 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:59.250410 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:59.253123 (+ 2713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:53:59.253129 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:53:59.253143 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:53:59.253173 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:53:59.254945 (+ 1772us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:53:59.255356 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:53:59.255361 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:53:59.255365 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:53:59.255473 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:53:59.255770 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:53:59.255773 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:53:59.255774 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:53:59.256294 (+ 520us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:53:59.256455 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0929 17:53:59.256788 (+ 333us) server_negotiation.cc:299] Negotiation successful
0929 17:53:59.257185 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":258,"thread_start_us":109,"threads_started":1}
I20260929 17:53:59.260462 10696 raft_consensus.cc:348] T 00000000000000000000000000000000 P c1225753a4f646bb8ac4cdcf73becc3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1225753a4f646bb8ac4cdcf73becc3b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35235 } }
I20260929 17:53:59.260680 10696 raft_consensus.cc:374] T 00000000000000000000000000000000 P c1225753a4f646bb8ac4cdcf73becc3b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:53:59.260716 10696 raft_consensus.cc:729] T 00000000000000000000000000000000 P c1225753a4f646bb8ac4cdcf73becc3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1225753a4f646bb8ac4cdcf73becc3b, State: Initialized, Role: FOLLOWER
I20260929 17:53:59.261166 10696 consensus_queue.cc:260] T 00000000000000000000000000000000 P c1225753a4f646bb8ac4cdcf73becc3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1225753a4f646bb8ac4cdcf73becc3b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35235 } }
I20260929 17:53:59.261292 10696 raft_consensus.cc:388] T 00000000000000000000000000000000 P c1225753a4f646bb8ac4cdcf73becc3b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 17:53:59.261334 10696 raft_consensus.cc:482] T 00000000000000000000000000000000 P c1225753a4f646bb8ac4cdcf73becc3b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 17:53:59.261425 10696 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c1225753a4f646bb8ac4cdcf73becc3b [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:53:59.262295 10696 raft_consensus.cc:504] T 00000000000000000000000000000000 P c1225753a4f646bb8ac4cdcf73becc3b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1225753a4f646bb8ac4cdcf73becc3b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35235 } }
I20260929 17:53:59.262629 10696 leader_election.cc:302] T 00000000000000000000000000000000 P c1225753a4f646bb8ac4cdcf73becc3b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c1225753a4f646bb8ac4cdcf73becc3b; no voters:
I20260929 17:53:59.262948 10696 leader_election.cc:288] T 00000000000000000000000000000000 P c1225753a4f646bb8ac4cdcf73becc3b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 17:53:59.263047 10701 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c1225753a4f646bb8ac4cdcf73becc3b [term 1 FOLLOWER]: Leader election won for term 1
I20260929 17:53:59.263284 10701 raft_consensus.cc:686] T 00000000000000000000000000000000 P c1225753a4f646bb8ac4cdcf73becc3b [term 1 LEADER]: Becoming Leader. State: Replica: c1225753a4f646bb8ac4cdcf73becc3b, State: Running, Role: LEADER
I20260929 17:53:59.263654 10701 consensus_queue.cc:237] T 00000000000000000000000000000000 P c1225753a4f646bb8ac4cdcf73becc3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1225753a4f646bb8ac4cdcf73becc3b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35235 } }
I20260929 17:53:59.264120 10696 sys_catalog.cc:564] T 00000000000000000000000000000000 P c1225753a4f646bb8ac4cdcf73becc3b [sys.catalog]: configured and running, proceeding with master startup.
I20260929 17:53:59.265281 10702 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1225753a4f646bb8ac4cdcf73becc3b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c1225753a4f646bb8ac4cdcf73becc3b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1225753a4f646bb8ac4cdcf73becc3b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35235 } } }
I20260929 17:53:59.265404 10702 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1225753a4f646bb8ac4cdcf73becc3b [sys.catalog]: This master's current role is: LEADER
I20260929 17:53:59.265671 10703 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1225753a4f646bb8ac4cdcf73becc3b [sys.catalog]: SysCatalogTable state changed. Reason: New leader c1225753a4f646bb8ac4cdcf73becc3b. Latest consensus state: current_term: 1 leader_uuid: "c1225753a4f646bb8ac4cdcf73becc3b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1225753a4f646bb8ac4cdcf73becc3b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35235 } } }
I20260929 17:53:59.265754 10703 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1225753a4f646bb8ac4cdcf73becc3b [sys.catalog]: This master's current role is: LEADER
I20260929 17:53:59.265791 10710 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 17:53:59.268875 10710 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 17:53:59.274225 10710 catalog_manager.cc:1347] Generated new cluster ID: ee75849c5314438ab83fee7e35ad7eab
I20260929 17:53:59.274297 10710 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 17:53:59.286829 10710 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 17:53:59.288064 10710 catalog_manager.cc:1504] Loading token signing keys...
I20260929 17:53:59.297107 10710 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c1225753a4f646bb8ac4cdcf73becc3b: Generated new TSK 0
I20260929 17:53:59.297955 10710 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704375975685-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704375975685-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260929 17:53:59.336781 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskup3Soe/build/debug/bin/kudu
/tmp/dist-test-taskup3Soe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704375975685-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704375975685-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704375975685-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704375975685-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35235
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704375975685-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704375975685-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704375975685-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704375975685-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:53:59.445626 10722 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:53:59.445891 10722 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:53:59.445935 10722 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:53:59.450002 10722 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:53:59.450064 10722 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:53:59.450136 10722 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 17:53:59.455379 10722 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704375975685-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704375975685-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704375975685-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704375975685-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35235
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10722
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704375975685-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:53:59.456390 10722 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:53:59.457442 10722 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:53:59.466892 10727 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:53:59.466914 10730 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:53:59.466976 10722 server_base.cc:1034] running on GCE node
W20260929 17:53:59.466912 10728 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:53:59.467491 10722 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:53:59.468147 10722 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:53:59.469306 10722 hybrid_clock.cc:648] HybridClock initialized: now 1790704439469285 us; error 42 us; skew 500 ppm
Sep 29 17:53:59 dist-test-slave-53nh krb5kdc[10615](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704439, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:53:59.472069 10722 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260929 17:53:59.472985 10722 webserver.cc:460] Webserver started at http://127.0.98.1:38051/ using document root <none> and password file <none>
I20260929 17:53:59.473564 10722 fs_manager.cc:359] Metadata directory not provided
I20260929 17:53:59.473662 10722 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:53:59.473870 10722 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:53:59.475906 10722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704375975685-392-0/minicluster-data/ts-0/data/instance:
uuid: "68617a4bfcea4a818f4856ac453a3cd4"
format_stamp: "Formatted at 2026-09-29 17:53:59 on dist-test-slave-53nh"
server_key: "e93bcb2a2839b9354e58f23ee9e5cb9f"
server_key_iv: "44df628a2bd6c195ca4ea6d1b61d747d"
server_key_version: "encryptionkey@0"
I20260929 17:53:59.476431 10722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704375975685-392-0/minicluster-data/ts-0/wal/instance:
uuid: "68617a4bfcea4a818f4856ac453a3cd4"
format_stamp: "Formatted at 2026-09-29 17:53:59 on dist-test-slave-53nh"
server_key: "e93bcb2a2839b9354e58f23ee9e5cb9f"
server_key_iv: "44df628a2bd6c195ca4ea6d1b61d747d"
server_key_version: "encryptionkey@0"
I20260929 17:53:59.480124 10722 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260929 17:53:59.482623 10737 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:53:59.483970 10722 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 17:53:59.484092 10722 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704375975685-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704375975685-392-0/minicluster-data/ts-0/wal
uuid: "68617a4bfcea4a818f4856ac453a3cd4"
format_stamp: "Formatted at 2026-09-29 17:53:59 on dist-test-slave-53nh"
server_key: "e93bcb2a2839b9354e58f23ee9e5cb9f"
server_key_iv: "44df628a2bd6c195ca4ea6d1b61d747d"
server_key_version: "encryptionkey@0"
I20260929 17:53:59.484200 10722 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704375975685-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704375975685-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704375975685-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:53:59.501741 10722 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:53:59.504619 10722 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:53:59.504788 10722 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:53:59.505393 10722 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 17:53:59.506201 10722 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:53:59.506246 10722 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:53:59.506296 10722 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:53:59.506325 10722 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 17:53:59.514441 10722 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45175
I20260929 17:53:59.514468 10850 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45175 every 8 connection(s)
I20260929 17:53:59.515288 10722 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704375975685-392-0/minicluster-data/ts-0/data/info.pb
I20260929 17:53:59.522863 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskup3Soe/build/debug/bin/kudu as pid 10722
I20260929 17:53:59.522975 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704375975685-392-0/minicluster-data/ts-0/wal/instance
I20260929 17:53:59.523288 392 external_mini_cluster.cc:1428] Setting key c311e1000213931f6472d814c3cfe1b5
Sep 29 17:53:59 dist-test-slave-53nh krb5kdc[10615](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704439, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 17:53:59.528733 10699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:59.517092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35691 (local address 127.0.98.62:35235)
0929 17:53:59.517222 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:59.517226 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:59.518035 (+ 809us) server_negotiation.cc:407] Connection header received
0929 17:53:59.519321 (+ 1286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:59.519324 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:59.519370 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:59.519447 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:53:59.520585 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:59.521224 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:59.522305 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:59.522533 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:59.522621 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:59.525267 (+ 2646us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:53:59.525271 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:53:59.525274 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:53:59.525322 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:53:59.526670 (+ 1348us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:53:59.527238 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:53:59.527241 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:53:59.527243 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:53:59.527318 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:53:59.527637 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:53:59.527641 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:53:59.527642 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:53:59.527869 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:53:59.527958 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0929 17:53:59.528434 (+ 476us) server_negotiation.cc:299] Negotiation successful
0929 17:53:59.528568 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 17:53:59.529556 10853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:59.517446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35235 (local address 127.0.98.1:35691)
0929 17:53:59.517892 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:59.517934 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 17:53:59.519092 (+ 1158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:53:59.519596 (+ 504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:53:59.519602 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:53:59.519935 (+ 333us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:53:59.520442 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:59.520454 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:59.521319 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:59.521322 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:59.522202 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:59.522210 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:59.522705 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:59.522710 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:59.522906 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:59.523572 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:53:59.523592 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:53:59.525129 (+ 1537us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:53:59.526789 (+ 1660us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:53:59.526795 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:53:59.526810 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:53:59.527140 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:53:59.527400 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:53:59.527402 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:53:59.527403 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:53:59.527541 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:53:59.527953 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:53:59.527958 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:53:59.528211 (+ 253us) client_negotiation.cc:769] Sending connection context
0929 17:53:59.528370 (+ 159us) client_negotiation.cc:240] Negotiation successful
0929 17:53:59.528644 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":116,"threads_started":1}
I20260929 17:53:59.530718 10851 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35235
I20260929 17:53:59.530917 10851 heartbeater.cc:464] Registering TS with master...
I20260929 17:53:59.531387 10851 heartbeater.cc:511] Master 127.0.98.62:35235 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260929 17:53:59.532910 10660 ts_manager.cc:194] Registered new tserver with Master: 68617a4bfcea4a818f4856ac453a3cd4 (127.0.98.1:45175)
I20260929 17:53:59.534026 10660 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:35691
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704375975685-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704375975685-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260929 17:53:59.549804 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskup3Soe/build/debug/bin/kudu
/tmp/dist-test-taskup3Soe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704375975685-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704375975685-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704375975685-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704375975685-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35235
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704375975685-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704375975685-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704375975685-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704375975685-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:53:59.657828 10856 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:53:59.658114 10856 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:53:59.658156 10856 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:53:59.662070 10856 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:53:59.662127 10856 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:53:59.662184 10856 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 17:53:59.667611 10856 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704375975685-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704375975685-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704375975685-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704375975685-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35235
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10856
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704375975685-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:53:59.668618 10856 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:53:59.669655 10856 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:53:59.678821 10862 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:53:59.678845 10861 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:53:59.679018 10856 server_base.cc:1034] running on GCE node
W20260929 17:53:59.678896 10864 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:53:59.679455 10856 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:53:59.680042 10856 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:53:59.681206 10856 hybrid_clock.cc:648] HybridClock initialized: now 1790704439681185 us; error 35 us; skew 500 ppm
Sep 29 17:53:59 dist-test-slave-53nh krb5kdc[10615](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704439, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:53:59.683771 10856 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260929 17:53:59.684589 10856 webserver.cc:460] Webserver started at http://127.0.98.2:36141/ using document root <none> and password file <none>
I20260929 17:53:59.685174 10856 fs_manager.cc:359] Metadata directory not provided
I20260929 17:53:59.685241 10856 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:53:59.685442 10856 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:53:59.687373 10856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704375975685-392-0/minicluster-data/ts-1/data/instance:
uuid: "5f51e69ac560469098e414a536ede5a0"
format_stamp: "Formatted at 2026-09-29 17:53:59 on dist-test-slave-53nh"
server_key: "2bb178f3d6f939c43078a2bc35b4f5dc"
server_key_iv: "da42c34fe7d658bce4342de6c461f359"
server_key_version: "encryptionkey@0"
I20260929 17:53:59.687886 10856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704375975685-392-0/minicluster-data/ts-1/wal/instance:
uuid: "5f51e69ac560469098e414a536ede5a0"
format_stamp: "Formatted at 2026-09-29 17:53:59 on dist-test-slave-53nh"
server_key: "2bb178f3d6f939c43078a2bc35b4f5dc"
server_key_iv: "da42c34fe7d658bce4342de6c461f359"
server_key_version: "encryptionkey@0"
I20260929 17:53:59.691403 10856 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260929 17:53:59.693789 10871 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:53:59.695024 10856 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260929 17:53:59.695123 10856 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704375975685-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704375975685-392-0/minicluster-data/ts-1/wal
uuid: "5f51e69ac560469098e414a536ede5a0"
format_stamp: "Formatted at 2026-09-29 17:53:59 on dist-test-slave-53nh"
server_key: "2bb178f3d6f939c43078a2bc35b4f5dc"
server_key_iv: "da42c34fe7d658bce4342de6c461f359"
server_key_version: "encryptionkey@0"
I20260929 17:53:59.695211 10856 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704375975685-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704375975685-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704375975685-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:53:59.710732 10856 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:53:59.713876 10856 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:53:59.714051 10856 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:53:59.714650 10856 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 17:53:59.715497 10856 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:53:59.715551 10856 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:53:59.715606 10856 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:53:59.715631 10856 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 17:53:59.724261 10856 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45499
I20260929 17:53:59.724285 10984 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45499 every 8 connection(s)
I20260929 17:53:59.725283 10856 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704375975685-392-0/minicluster-data/ts-1/data/info.pb
I20260929 17:53:59.725561 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskup3Soe/build/debug/bin/kudu as pid 10856
I20260929 17:53:59.725672 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704375975685-392-0/minicluster-data/ts-1/wal/instance
I20260929 17:53:59.725982 392 external_mini_cluster.cc:1428] Setting key 019b52d9fcd313ee1a5288961f9edff6
Sep 29 17:53:59 dist-test-slave-53nh krb5kdc[10615](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704439, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260929 17:53:59.739867 10699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:59.727248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48761 (local address 127.0.98.62:35235)
0929 17:53:59.727360 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:59.727365 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:59.728481 (+ 1116us) server_negotiation.cc:407] Connection header received
0929 17:53:59.730075 (+ 1594us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:59.730078 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:59.730128 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:59.730204 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:53:59.731438 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:59.731980 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:59.733260 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:59.733490 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:59.733579 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:59.736379 (+ 2800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:53:59.736383 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:53:59.736385 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:53:59.736410 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:53:59.737874 (+ 1464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:53:59.738394 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:53:59.738397 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:53:59.738398 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:53:59.738472 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:53:59.738780 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:53:59.738783 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:53:59.738785 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:53:59.738963 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:53:59.739044 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0929 17:53:59.739582 (+ 538us) server_negotiation.cc:299] Negotiation successful
0929 17:53:59.739704 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260929 17:53:59.740783 10987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:59.727772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35235 (local address 127.0.98.2:48761)
0929 17:53:59.728344 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:59.728388 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 17:53:59.729813 (+ 1425us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:53:59.730414 (+ 601us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:53:59.730421 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:53:59.730796 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:53:59.731291 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:59.731304 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:59.732163 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:59.732166 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:59.733146 (+ 980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:59.733156 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:59.733639 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:59.733643 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:59.733827 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:59.734487 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:53:59.734505 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:53:59.736161 (+ 1656us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:53:59.737987 (+ 1826us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:53:59.737993 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:53:59.738008 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:53:59.738300 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:53:59.738553 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:53:59.738562 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:53:59.738563 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:53:59.738695 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:53:59.739068 (+ 373us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:53:59.739074 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:53:59.739348 (+ 274us) client_negotiation.cc:769] Sending connection context
0929 17:53:59.739521 (+ 173us) client_negotiation.cc:240] Negotiation successful
0929 17:53:59.739854 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":155,"threads_started":1}
I20260929 17:53:59.742336 10985 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35235
I20260929 17:53:59.742596 10985 heartbeater.cc:464] Registering TS with master...
I20260929 17:53:59.743153 10985 heartbeater.cc:511] Master 127.0.98.62:35235 requested a full tablet report, sending...
I20260929 17:53:59.744318 10660 ts_manager.cc:194] Registered new tserver with Master: 5f51e69ac560469098e414a536ede5a0 (127.0.98.2:45499)
I20260929 17:53:59.745160 10660 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:48761
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704375975685-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704375975685-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260929 17:53:59.757970 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskup3Soe/build/debug/bin/kudu
/tmp/dist-test-taskup3Soe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704375975685-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704375975685-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704375975685-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704375975685-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35235
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704375975685-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704375975685-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704375975685-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704375975685-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:53:59.865469 10990 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:53:59.865725 10990 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:53:59.865770 10990 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:53:59.869805 10990 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:53:59.869865 10990 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:53:59.869922 10990 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 17:53:59.875160 10990 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704375975685-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704375975685-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704375975685-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704375975685-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35235
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10990
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704375975685-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:53:59.876194 10990 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:53:59.877213 10990 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:53:59.886785 10998 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:53:59.886790 10996 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:53:59.886790 10995 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:53:59.886857 10990 server_base.cc:1034] running on GCE node
I20260929 17:53:59.887389 10990 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:53:59.888037 10990 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:53:59.889191 10990 hybrid_clock.cc:648] HybridClock initialized: now 1790704439889173 us; error 33 us; skew 500 ppm
Sep 29 17:53:59 dist-test-slave-53nh krb5kdc[10615](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704439, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:53:59.891708 10990 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260929 17:53:59.892524 10990 webserver.cc:460] Webserver started at http://127.0.98.3:42719/ using document root <none> and password file <none>
I20260929 17:53:59.893069 10990 fs_manager.cc:359] Metadata directory not provided
I20260929 17:53:59.893139 10990 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:53:59.893332 10990 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:53:59.895262 10990 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704375975685-392-0/minicluster-data/ts-2/data/instance:
uuid: "de2522023d1242cbbf12ba4fb91e5549"
format_stamp: "Formatted at 2026-09-29 17:53:59 on dist-test-slave-53nh"
server_key: "dd65760f32e418d460946d3cfe6591d1"
server_key_iv: "d71d32d8111b35eb126ed520b5522fd8"
server_key_version: "encryptionkey@0"
I20260929 17:53:59.895783 10990 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704375975685-392-0/minicluster-data/ts-2/wal/instance:
uuid: "de2522023d1242cbbf12ba4fb91e5549"
format_stamp: "Formatted at 2026-09-29 17:53:59 on dist-test-slave-53nh"
server_key: "dd65760f32e418d460946d3cfe6591d1"
server_key_iv: "d71d32d8111b35eb126ed520b5522fd8"
server_key_version: "encryptionkey@0"
I20260929 17:53:59.899392 10990 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 17:53:59.901834 11005 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:53:59.902997 10990 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 17:53:59.903102 10990 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704375975685-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704375975685-392-0/minicluster-data/ts-2/wal
uuid: "de2522023d1242cbbf12ba4fb91e5549"
format_stamp: "Formatted at 2026-09-29 17:53:59 on dist-test-slave-53nh"
server_key: "dd65760f32e418d460946d3cfe6591d1"
server_key_iv: "d71d32d8111b35eb126ed520b5522fd8"
server_key_version: "encryptionkey@0"
I20260929 17:53:59.903218 10990 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704375975685-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704375975685-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704375975685-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:53:59.917486 10990 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:53:59.920351 10990 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:53:59.920524 10990 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:53:59.921144 10990 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 17:53:59.921918 10990 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:53:59.921965 10990 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:53:59.922003 10990 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:53:59.922039 10990 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 17:53:59.930308 10990 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46615
I20260929 17:53:59.930334 11118 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46615 every 8 connection(s)
I20260929 17:53:59.931161 10990 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704375975685-392-0/minicluster-data/ts-2/data/info.pb
I20260929 17:53:59.933848 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskup3Soe/build/debug/bin/kudu as pid 10990
I20260929 17:53:59.933960 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704375975685-392-0/minicluster-data/ts-2/wal/instance
I20260929 17:53:59.934221 392 external_mini_cluster.cc:1428] Setting key f74f5c2518ce32fe4abe4716d44fbbfb
Sep 29 17:53:59 dist-test-slave-53nh krb5kdc[10615](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704439, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 17:53:59.944187 10699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:59.932791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36059 (local address 127.0.98.62:35235)
0929 17:53:59.932918 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:59.932921 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:59.933827 (+ 906us) server_negotiation.cc:407] Connection header received
0929 17:53:59.935162 (+ 1335us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:59.935165 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:59.935218 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:59.935289 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:53:59.936579 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:59.937144 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:59.938238 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:59.938452 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:59.938535 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:59.940801 (+ 2266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:53:59.940804 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:53:59.940807 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:53:59.940829 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:53:59.942217 (+ 1388us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:53:59.942723 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:53:59.942726 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:53:59.942727 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:53:59.942802 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:53:59.943105 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:53:59.943108 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:53:59.943110 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:53:59.943293 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:53:59.943393 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0929 17:53:59.943886 (+ 493us) server_negotiation.cc:299] Negotiation successful
0929 17:53:59.944008 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 17:53:59.944990 11121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:59.933170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35235 (local address 127.0.98.3:36059)
0929 17:53:59.933672 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:59.933713 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 17:53:59.934974 (+ 1261us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:53:59.935467 (+ 493us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:53:59.935477 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:53:59.935949 (+ 472us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:53:59.936448 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:59.936461 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:59.937258 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:59.937261 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:59.938129 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:59.938138 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:59.938584 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:59.938588 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:59.938745 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:59.939260 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:53:59.939276 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:53:59.940656 (+ 1380us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:53:59.942339 (+ 1683us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:53:59.942344 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:53:59.942359 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:53:59.942630 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:53:59.942881 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:53:59.942884 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:53:59.942886 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:53:59.943016 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:53:59.943389 (+ 373us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:53:59.943395 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:53:59.943649 (+ 254us) client_negotiation.cc:769] Sending connection context
0929 17:53:59.943827 (+ 178us) client_negotiation.cc:240] Negotiation successful
0929 17:53:59.944164 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":124,"threads_started":1}
I20260929 17:53:59.946192 11119 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35235
I20260929 17:53:59.946416 11119 heartbeater.cc:464] Registering TS with master...
I20260929 17:53:59.946898 11119 heartbeater.cc:511] Master 127.0.98.62:35235 requested a full tablet report, sending...
I20260929 17:53:59.947962 10660 ts_manager.cc:194] Registered new tserver with Master: de2522023d1242cbbf12ba4fb91e5549 (127.0.98.3:46615)
I20260929 17:53:59.948499 10660 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:36059
I20260929 17:53:59.959632 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 17:53:59.968567 10699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:59.961472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53784 (local address 127.0.98.62:35235)
0929 17:53:59.961608 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:59.961612 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:59.961781 (+ 169us) server_negotiation.cc:407] Connection header received
0929 17:53:59.961893 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:59.961896 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:59.961939 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:59.962000 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:53:59.962673 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:59.963196 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:59.964072 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:59.964277 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:59.964359 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:59.965881 (+ 1522us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:53:59.965885 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:53:59.965887 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:53:59.965911 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:53:59.967038 (+ 1127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:53:59.967420 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:53:59.967423 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:53:59.967425 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:53:59.967497 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:53:59.967748 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:53:59.967752 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:53:59.967753 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:53:59.967966 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:53:59.968083 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0929 17:53:59.968301 (+ 218us) server_negotiation.cc:299] Negotiation successful
0929 17:53:59.968406 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 17:53:59.971212 10660 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53784:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 17:53:59.974188 10660 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 17:53:59.987394 11130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:59.982023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45499 (local address 127.0.0.1:33998)
0929 17:53:59.982586 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:59.982634 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0929 17:53:59.982755 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:53:59.983279 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:53:59.983285 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:53:59.983310 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:53:59.983442 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:59.983449 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:59.984771 (+ 1322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:59.984774 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:59.985740 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:59.985748 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:59.986862 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:59.986866 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:59.986994 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:59.987032 (+ 38us) client_negotiation.cc:769] Sending connection context
0929 17:53:59.987128 (+ 96us) client_negotiation.cc:240] Negotiation successful
0929 17:53:59.987246 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":396,"mutex_wait_us":18,"thread_start_us":88,"threads_started":1}
I20260929 17:53:59.987402 11132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:59.982430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33998 (local address 127.0.98.2:45499)
0929 17:53:59.982764 (+ 334us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:59.982777 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:59.982802 (+ 25us) server_negotiation.cc:407] Connection header received
0929 17:53:59.982883 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:59.982888 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:59.983116 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:59.983276 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:53:59.983600 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:59.984668 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:59.985853 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:59.986731 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:59.986971 (+ 240us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:59.987141 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0929 17:53:59.987190 (+ 49us) server_negotiation.cc:299] Negotiation successful
0929 17:53:59.987274 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":242,"thread_start_us":120,"threads_started":1}
I20260929 17:53:59.988220 11134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:59.982147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45175 (local address 127.0.0.1:33080)
0929 17:53:59.983144 (+ 997us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:59.983157 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 17:53:59.983231 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:53:59.984098 (+ 867us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:53:59.984101 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:53:59.984119 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:53:59.984232 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:59.984239 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:59.985740 (+ 1501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:59.985743 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:59.986844 (+ 1101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:59.986853 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:59.987874 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:59.987878 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:59.987952 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:59.987969 (+ 17us) client_negotiation.cc:769] Sending connection context
0929 17:53:59.988030 (+ 61us) client_negotiation.cc:240] Negotiation successful
0929 17:53:59.988097 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":940,"thread_start_us":103,"threads_started":1}
I20260929 17:53:59.988366 11133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:59.982733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33080 (local address 127.0.98.1:45175)
0929 17:53:59.983652 (+ 919us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:59.983665 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:59.983684 (+ 19us) server_negotiation.cc:407] Connection header received
0929 17:53:59.983766 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:59.983771 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:59.983942 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:59.984092 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:53:59.984366 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:59.985579 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:59.986966 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:59.987593 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:59.987685 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:59.987802 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0929 17:53:59.988079 (+ 277us) server_negotiation.cc:299] Negotiation successful
0929 17:53:59.988235 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":824,"thread_start_us":119,"threads_started":1}
I20260929 17:53:59.989250 11131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:59.982866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46615 (local address 127.0.0.1:48602)
0929 17:53:59.983554 (+ 688us) negotiation.cc:107] Waiting for socket to connect
0929 17:53:59.983566 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 17:53:59.983833 (+ 267us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:53:59.984305 (+ 472us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:53:59.984308 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:53:59.984325 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:53:59.984411 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:59.984416 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:59.985772 (+ 1356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:59.985774 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:59.987656 (+ 1882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:53:59.987667 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:53:59.988862 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:53:59.988866 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:53:59.988948 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:59.988970 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 17:53:59.989070 (+ 100us) client_negotiation.cc:240] Negotiation successful
0929 17:53:59.989134 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":617,"mutex_wait_us":3,"thread_start_us":85,"threads_started":1}
I20260929 17:53:59.989988 11135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:53:59.983086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48602 (local address 127.0.98.3:46615)
0929 17:53:59.983462 (+ 376us) server_negotiation.cc:206] Beginning negotiation
0929 17:53:59.983474 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 17:53:59.983700 (+ 226us) server_negotiation.cc:407] Connection header received
0929 17:53:59.983986 (+ 286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:53:59.983992 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:53:59.984166 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:53:59.984321 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:53:59.984500 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:59.985634 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:59.987821 (+ 2187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:53:59.988565 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:53:59.988661 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:53:59.988803 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0929 17:53:59.989756 (+ 953us) server_negotiation.cc:299] Negotiation successful
0929 17:53:59.989853 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":289,"thread_start_us":94,"threads_started":1}
I20260929 17:53:59.990839 10919 tablet_service.cc:1467] Processing CreateTablet for tablet a36815a14c294d7888a69fabf4c66fa2 (DEFAULT_TABLE table=test-table [id=e724fc0a726641c1b0e57f583d0af58b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 17:53:59.990850 10785 tablet_service.cc:1467] Processing CreateTablet for tablet a36815a14c294d7888a69fabf4c66fa2 (DEFAULT_TABLE table=test-table [id=e724fc0a726641c1b0e57f583d0af58b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 17:53:59.991801 10919 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a36815a14c294d7888a69fabf4c66fa2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:53:59.991838 10785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a36815a14c294d7888a69fabf4c66fa2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:53:59.992622 11053 tablet_service.cc:1467] Processing CreateTablet for tablet a36815a14c294d7888a69fabf4c66fa2 (DEFAULT_TABLE table=test-table [id=e724fc0a726641c1b0e57f583d0af58b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 17:53:59.993531 11053 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a36815a14c294d7888a69fabf4c66fa2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:53:59.997944 11136 tablet_bootstrap.cc:492] T a36815a14c294d7888a69fabf4c66fa2 P 5f51e69ac560469098e414a536ede5a0: Bootstrap starting.
I20260929 17:53:59.999699 11137 tablet_bootstrap.cc:492] T a36815a14c294d7888a69fabf4c66fa2 P 68617a4bfcea4a818f4856ac453a3cd4: Bootstrap starting.
I20260929 17:54:00.000169 11138 tablet_bootstrap.cc:492] T a36815a14c294d7888a69fabf4c66fa2 P de2522023d1242cbbf12ba4fb91e5549: Bootstrap starting.
I20260929 17:54:00.001962 11136 tablet_bootstrap.cc:654] T a36815a14c294d7888a69fabf4c66fa2 P 5f51e69ac560469098e414a536ede5a0: Neither blocks nor log segments found. Creating new log.
I20260929 17:54:00.002069 11137 tablet_bootstrap.cc:654] T a36815a14c294d7888a69fabf4c66fa2 P 68617a4bfcea4a818f4856ac453a3cd4: Neither blocks nor log segments found. Creating new log.
I20260929 17:54:00.002436 11138 tablet_bootstrap.cc:654] T a36815a14c294d7888a69fabf4c66fa2 P de2522023d1242cbbf12ba4fb91e5549: Neither blocks nor log segments found. Creating new log.
I20260929 17:54:00.002938 11136 log.cc:824] T a36815a14c294d7888a69fabf4c66fa2 P 5f51e69ac560469098e414a536ede5a0: Log is configured to *not* fsync() on all Append() calls
I20260929 17:54:00.003196 11137 log.cc:824] T a36815a14c294d7888a69fabf4c66fa2 P 68617a4bfcea4a818f4856ac453a3cd4: Log is configured to *not* fsync() on all Append() calls
I20260929 17:54:00.003453 11138 log.cc:824] T a36815a14c294d7888a69fabf4c66fa2 P de2522023d1242cbbf12ba4fb91e5549: Log is configured to *not* fsync() on all Append() calls
I20260929 17:54:00.005633 11138 tablet_bootstrap.cc:492] T a36815a14c294d7888a69fabf4c66fa2 P de2522023d1242cbbf12ba4fb91e5549: No bootstrap required, opened a new log
I20260929 17:54:00.005666 11136 tablet_bootstrap.cc:492] T a36815a14c294d7888a69fabf4c66fa2 P 5f51e69ac560469098e414a536ede5a0: No bootstrap required, opened a new log
I20260929 17:54:00.005681 11137 tablet_bootstrap.cc:492] T a36815a14c294d7888a69fabf4c66fa2 P 68617a4bfcea4a818f4856ac453a3cd4: No bootstrap required, opened a new log
I20260929 17:54:00.005847 11138 ts_tablet_manager.cc:1397] T a36815a14c294d7888a69fabf4c66fa2 P de2522023d1242cbbf12ba4fb91e5549: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260929 17:54:00.005851 11137 ts_tablet_manager.cc:1397] T a36815a14c294d7888a69fabf4c66fa2 P 68617a4bfcea4a818f4856ac453a3cd4: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260929 17:54:00.005851 11136 ts_tablet_manager.cc:1397] T a36815a14c294d7888a69fabf4c66fa2 P 5f51e69ac560469098e414a536ede5a0: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.001s
I20260929 17:54:00.009102 11138 raft_consensus.cc:348] T a36815a14c294d7888a69fabf4c66fa2 P de2522023d1242cbbf12ba4fb91e5549 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f51e69ac560469098e414a536ede5a0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45499 } } peers { permanent_uuid: "68617a4bfcea4a818f4856ac453a3cd4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45175 } } peers { permanent_uuid: "de2522023d1242cbbf12ba4fb91e5549" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46615 } }
I20260929 17:54:00.009361 11138 raft_consensus.cc:374] T a36815a14c294d7888a69fabf4c66fa2 P de2522023d1242cbbf12ba4fb91e5549 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:54:00.009433 11138 raft_consensus.cc:729] T a36815a14c294d7888a69fabf4c66fa2 P de2522023d1242cbbf12ba4fb91e5549 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de2522023d1242cbbf12ba4fb91e5549, State: Initialized, Role: FOLLOWER
I20260929 17:54:00.009867 11138 consensus_queue.cc:260] T a36815a14c294d7888a69fabf4c66fa2 P de2522023d1242cbbf12ba4fb91e5549 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f51e69ac560469098e414a536ede5a0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45499 } } peers { permanent_uuid: "68617a4bfcea4a818f4856ac453a3cd4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45175 } } peers { permanent_uuid: "de2522023d1242cbbf12ba4fb91e5549" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46615 } }
I20260929 17:54:00.009737 11137 raft_consensus.cc:348] T a36815a14c294d7888a69fabf4c66fa2 P 68617a4bfcea4a818f4856ac453a3cd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f51e69ac560469098e414a536ede5a0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45499 } } peers { permanent_uuid: "68617a4bfcea4a818f4856ac453a3cd4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45175 } } peers { permanent_uuid: "de2522023d1242cbbf12ba4fb91e5549" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46615 } }
I20260929 17:54:00.009737 11136 raft_consensus.cc:348] T a36815a14c294d7888a69fabf4c66fa2 P 5f51e69ac560469098e414a536ede5a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f51e69ac560469098e414a536ede5a0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45499 } } peers { permanent_uuid: "68617a4bfcea4a818f4856ac453a3cd4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45175 } } peers { permanent_uuid: "de2522023d1242cbbf12ba4fb91e5549" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46615 } }
I20260929 17:54:00.009989 11137 raft_consensus.cc:374] T a36815a14c294d7888a69fabf4c66fa2 P 68617a4bfcea4a818f4856ac453a3cd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:54:00.009989 11136 raft_consensus.cc:374] T a36815a14c294d7888a69fabf4c66fa2 P 5f51e69ac560469098e414a536ede5a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:54:00.010048 11136 raft_consensus.cc:729] T a36815a14c294d7888a69fabf4c66fa2 P 5f51e69ac560469098e414a536ede5a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f51e69ac560469098e414a536ede5a0, State: Initialized, Role: FOLLOWER
I20260929 17:54:00.010080 11137 raft_consensus.cc:729] T a36815a14c294d7888a69fabf4c66fa2 P 68617a4bfcea4a818f4856ac453a3cd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68617a4bfcea4a818f4856ac453a3cd4, State: Initialized, Role: FOLLOWER
I20260929 17:54:00.010545 11119 heartbeater.cc:503] Master 127.0.98.62:35235 was elected leader, sending a full tablet report...
I20260929 17:54:00.010545 11136 consensus_queue.cc:260] T a36815a14c294d7888a69fabf4c66fa2 P 5f51e69ac560469098e414a536ede5a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f51e69ac560469098e414a536ede5a0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45499 } } peers { permanent_uuid: "68617a4bfcea4a818f4856ac453a3cd4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45175 } } peers { permanent_uuid: "de2522023d1242cbbf12ba4fb91e5549" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46615 } }
I20260929 17:54:00.010547 11137 consensus_queue.cc:260] T a36815a14c294d7888a69fabf4c66fa2 P 68617a4bfcea4a818f4856ac453a3cd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f51e69ac560469098e414a536ede5a0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45499 } } peers { permanent_uuid: "68617a4bfcea4a818f4856ac453a3cd4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45175 } } peers { permanent_uuid: "de2522023d1242cbbf12ba4fb91e5549" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46615 } }
I20260929 17:54:00.010941 11138 ts_tablet_manager.cc:1428] T a36815a14c294d7888a69fabf4c66fa2 P de2522023d1242cbbf12ba4fb91e5549: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260929 17:54:00.011543 11136 ts_tablet_manager.cc:1428] T a36815a14c294d7888a69fabf4c66fa2 P 5f51e69ac560469098e414a536ede5a0: Time spent starting tablet: real 0.006s user 0.003s sys 0.003s
I20260929 17:54:00.011543 11137 ts_tablet_manager.cc:1428] T a36815a14c294d7888a69fabf4c66fa2 P 68617a4bfcea4a818f4856ac453a3cd4: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260929 17:54:00.011870 10985 heartbeater.cc:503] Master 127.0.98.62:35235 was elected leader, sending a full tablet report...
I20260929 17:54:00.012001 10851 heartbeater.cc:503] Master 127.0.98.62:35235 was elected leader, sending a full tablet report...
W20260929 17:54:00.016353 10852 tablet.cc:2369] T a36815a14c294d7888a69fabf4c66fa2 P 68617a4bfcea4a818f4856ac453a3cd4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 17:54:00.182125 11120 tablet.cc:2369] T a36815a14c294d7888a69fabf4c66fa2 P de2522023d1242cbbf12ba4fb91e5549: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 17:54:00.196019 11143 raft_consensus.cc:482] T a36815a14c294d7888a69fabf4c66fa2 P 5f51e69ac560469098e414a536ede5a0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 17:54:00.196245 11143 raft_consensus.cc:504] T a36815a14c294d7888a69fabf4c66fa2 P 5f51e69ac560469098e414a536ede5a0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f51e69ac560469098e414a536ede5a0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45499 } } peers { permanent_uuid: "68617a4bfcea4a818f4856ac453a3cd4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45175 } } peers { permanent_uuid: "de2522023d1242cbbf12ba4fb91e5549" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46615 } }
I20260929 17:54:00.197368 11143 leader_election.cc:288] T a36815a14c294d7888a69fabf4c66fa2 P 5f51e69ac560469098e414a536ede5a0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 68617a4bfcea4a818f4856ac453a3cd4 (127.0.98.1:45175), de2522023d1242cbbf12ba4fb91e5549 (127.0.98.3:46615)
I20260929 17:54:00.201990 10987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:54:00.197603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45175 (local address 127.0.98.2:57657)
0929 17:54:00.197862 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0929 17:54:00.197877 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 17:54:00.197987 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:54:00.198281 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:54:00.198285 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:54:00.198304 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:54:00.198428 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:54:00.198435 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:54:00.199459 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:54:00.199462 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:54:00.200719 (+ 1257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:54:00.200729 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:54:00.201633 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:54:00.201637 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:54:00.201721 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:54:00.201742 (+ 21us) client_negotiation.cc:769] Sending connection context
0929 17:54:00.201802 (+ 60us) client_negotiation.cc:240] Negotiation successful
0929 17:54:00.201862 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"mutex_wait_us":1}
I20260929 17:54:00.202036 11133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:54:00.197734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57657 (local address 127.0.98.1:45175)
0929 17:54:00.197875 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0929 17:54:00.197879 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:54:00.197949 (+ 70us) server_negotiation.cc:407] Connection header received
0929 17:54:00.198104 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:54:00.198108 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:54:00.198162 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:54:00.198235 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:54:00.198565 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:54:00.199367 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:54:00.200855 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:54:00.201496 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:54:00.201615 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:54:00.201676 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0929 17:54:00.201838 (+ 162us) server_negotiation.cc:299] Negotiation successful
0929 17:54:00.201912 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 17:54:00.202134 11145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:54:00.197603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46615 (local address 127.0.98.2:58939)
0929 17:54:00.198063 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0929 17:54:00.198077 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 17:54:00.198178 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:54:00.198476 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:54:00.198480 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:54:00.198498 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:54:00.198604 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:54:00.198609 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:54:00.199602 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:54:00.199605 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:54:00.200754 (+ 1149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:54:00.200761 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:54:00.201734 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:54:00.201737 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:54:00.201843 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:54:00.201865 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 17:54:00.201936 (+ 71us) client_negotiation.cc:240] Negotiation successful
0929 17:54:00.202002 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":397,"mutex_wait_us":93,"thread_start_us":101,"threads_started":1}
I20260929 17:54:00.202147 11135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:54:00.197736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58939 (local address 127.0.98.3:46615)
0929 17:54:00.197861 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0929 17:54:00.197865 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:54:00.198194 (+ 329us) server_negotiation.cc:407] Connection header received
0929 17:54:00.198288 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:54:00.198295 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:54:00.198346 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:54:00.198422 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:54:00.198731 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:54:00.199488 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:54:00.200865 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:54:00.201529 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:54:00.201614 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:54:00.201676 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0929 17:54:00.201969 (+ 293us) server_negotiation.cc:299] Negotiation successful
0929 17:54:00.202044 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 17:54:00.202664 10805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a36815a14c294d7888a69fabf4c66fa2" candidate_uuid: "5f51e69ac560469098e414a536ede5a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68617a4bfcea4a818f4856ac453a3cd4" is_pre_election: true
I20260929 17:54:00.202857 11073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a36815a14c294d7888a69fabf4c66fa2" candidate_uuid: "5f51e69ac560469098e414a536ede5a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de2522023d1242cbbf12ba4fb91e5549" is_pre_election: true
I20260929 17:54:00.202967 10805 raft_consensus.cc:2454] T a36815a14c294d7888a69fabf4c66fa2 P 68617a4bfcea4a818f4856ac453a3cd4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f51e69ac560469098e414a536ede5a0 in term 0.
I20260929 17:54:00.203138 11073 raft_consensus.cc:2454] T a36815a14c294d7888a69fabf4c66fa2 P de2522023d1242cbbf12ba4fb91e5549 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f51e69ac560469098e414a536ede5a0 in term 0.
I20260929 17:54:00.203430 10873 leader_election.cc:302] T a36815a14c294d7888a69fabf4c66fa2 P 5f51e69ac560469098e414a536ede5a0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f51e69ac560469098e414a536ede5a0, 68617a4bfcea4a818f4856ac453a3cd4; no voters:
I20260929 17:54:00.203732 11143 raft_consensus.cc:2781] T a36815a14c294d7888a69fabf4c66fa2 P 5f51e69ac560469098e414a536ede5a0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 17:54:00.203806 11143 raft_consensus.cc:482] T a36815a14c294d7888a69fabf4c66fa2 P 5f51e69ac560469098e414a536ede5a0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 17:54:00.203838 11143 raft_consensus.cc:3037] T a36815a14c294d7888a69fabf4c66fa2 P 5f51e69ac560469098e414a536ede5a0 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:54:00.204840 11143 raft_consensus.cc:504] T a36815a14c294d7888a69fabf4c66fa2 P 5f51e69ac560469098e414a536ede5a0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f51e69ac560469098e414a536ede5a0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45499 } } peers { permanent_uuid: "68617a4bfcea4a818f4856ac453a3cd4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45175 } } peers { permanent_uuid: "de2522023d1242cbbf12ba4fb91e5549" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46615 } }
I20260929 17:54:00.205195 11143 leader_election.cc:288] T a36815a14c294d7888a69fabf4c66fa2 P 5f51e69ac560469098e414a536ede5a0 [CANDIDATE]: Term 1 election: Requested vote from peers 68617a4bfcea4a818f4856ac453a3cd4 (127.0.98.1:45175), de2522023d1242cbbf12ba4fb91e5549 (127.0.98.3:46615)
I20260929 17:54:00.205543 10805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a36815a14c294d7888a69fabf4c66fa2" candidate_uuid: "5f51e69ac560469098e414a536ede5a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68617a4bfcea4a818f4856ac453a3cd4"
I20260929 17:54:00.205549 11073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a36815a14c294d7888a69fabf4c66fa2" candidate_uuid: "5f51e69ac560469098e414a536ede5a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de2522023d1242cbbf12ba4fb91e5549"
I20260929 17:54:00.205665 10805 raft_consensus.cc:3037] T a36815a14c294d7888a69fabf4c66fa2 P 68617a4bfcea4a818f4856ac453a3cd4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:54:00.205665 11073 raft_consensus.cc:3037] T a36815a14c294d7888a69fabf4c66fa2 P de2522023d1242cbbf12ba4fb91e5549 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:54:00.206878 10805 raft_consensus.cc:2454] T a36815a14c294d7888a69fabf4c66fa2 P 68617a4bfcea4a818f4856ac453a3cd4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f51e69ac560469098e414a536ede5a0 in term 1.
I20260929 17:54:00.206878 11073 raft_consensus.cc:2454] T a36815a14c294d7888a69fabf4c66fa2 P de2522023d1242cbbf12ba4fb91e5549 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f51e69ac560469098e414a536ede5a0 in term 1.
I20260929 17:54:00.207166 10873 leader_election.cc:302] T a36815a14c294d7888a69fabf4c66fa2 P 5f51e69ac560469098e414a536ede5a0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 5f51e69ac560469098e414a536ede5a0, 68617a4bfcea4a818f4856ac453a3cd4, de2522023d1242cbbf12ba4fb91e5549; no voters:
I20260929 17:54:00.207329 11143 raft_consensus.cc:2781] T a36815a14c294d7888a69fabf4c66fa2 P 5f51e69ac560469098e414a536ede5a0 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 17:54:00.207549 11143 raft_consensus.cc:686] T a36815a14c294d7888a69fabf4c66fa2 P 5f51e69ac560469098e414a536ede5a0 [term 1 LEADER]: Becoming Leader. State: Replica: 5f51e69ac560469098e414a536ede5a0, State: Running, Role: LEADER
I20260929 17:54:00.207927 11143 consensus_queue.cc:237] T a36815a14c294d7888a69fabf4c66fa2 P 5f51e69ac560469098e414a536ede5a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f51e69ac560469098e414a536ede5a0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45499 } } peers { permanent_uuid: "68617a4bfcea4a818f4856ac453a3cd4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45175 } } peers { permanent_uuid: "de2522023d1242cbbf12ba4fb91e5549" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46615 } }
I20260929 17:54:00.210979 10659 catalog_manager.cc:5495] T a36815a14c294d7888a69fabf4c66fa2 P 5f51e69ac560469098e414a536ede5a0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5f51e69ac560469098e414a536ede5a0 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "5f51e69ac560469098e414a536ede5a0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f51e69ac560469098e414a536ede5a0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45499 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "68617a4bfcea4a818f4856ac453a3cd4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45175 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "de2522023d1242cbbf12ba4fb91e5549" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46615 } health_report { overall_health: UNKNOWN } } }
I20260929 17:54:00.223304 11132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:54:00.219853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34002 (local address 127.0.98.2:45499)
0929 17:54:00.220022 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0929 17:54:00.220026 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:54:00.220040 (+ 14us) server_negotiation.cc:407] Connection header received
0929 17:54:00.220113 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:54:00.220115 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:54:00.220175 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:54:00.220270 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 17:54:00.220502 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:54:00.221052 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:54:00.221894 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:54:00.222117 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:54:00.222224 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:54:00.222460 (+ 236us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 17:54:00.222869 (+ 409us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 17:54:00.222959 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0929 17:54:00.223105 (+ 146us) server_negotiation.cc:299] Negotiation successful
0929 17:54:00.223156 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
W20260929 17:54:00.226702 10986 tablet.cc:2369] T a36815a14c294d7888a69fabf4c66fa2 P 5f51e69ac560469098e414a536ede5a0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 17:54:00.229378 10805 raft_consensus.cc:1261] T a36815a14c294d7888a69fabf4c66fa2 P 68617a4bfcea4a818f4856ac453a3cd4 [term 1 FOLLOWER]: Refusing update from remote peer 5f51e69ac560469098e414a536ede5a0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 17:54:00.229377 11073 raft_consensus.cc:1261] T a36815a14c294d7888a69fabf4c66fa2 P de2522023d1242cbbf12ba4fb91e5549 [term 1 FOLLOWER]: Refusing update from remote peer 5f51e69ac560469098e414a536ede5a0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 17:54:00.230091 11146 consensus_queue.cc:1035] T a36815a14c294d7888a69fabf4c66fa2 P 5f51e69ac560469098e414a536ede5a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68617a4bfcea4a818f4856ac453a3cd4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 17:54:00.230293 11143 consensus_queue.cc:1035] T a36815a14c294d7888a69fabf4c66fa2 P 5f51e69ac560469098e414a536ede5a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "de2522023d1242cbbf12ba4fb91e5549" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 17:54:00.236056 11149 mvcc.cc:205] Tried to move back new op lower bound from 7334725387171971072 to 7334725387094114304. Current Snapshot: MvccSnapshot[applied={T|T < 7334725387171971072}]
I20260929 17:54:00.236563 11150 mvcc.cc:205] Tried to move back new op lower bound from 7334725387171971072 to 7334725387094114304. Current Snapshot: MvccSnapshot[applied={T|T < 7334725387171971072}]
I20260929 17:54:00.238497 11151 mvcc.cc:205] Tried to move back new op lower bound from 7334725387171971072 to 7334725387094114304. Current Snapshot: MvccSnapshot[applied={T|T < 7334725387171971072}]
I20260929 17:54:00.247476 10659 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53784:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 17:54:00.247653 10659 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53784:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 17:54:00.249969 10659 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c1225753a4f646bb8ac4cdcf73becc3b: Sending DeleteTablet for 3 replicas of tablet a36815a14c294d7888a69fabf4c66fa2
I20260929 17:54:00.250777 10919 tablet_service.cc:1514] Processing DeleteTablet for tablet a36815a14c294d7888a69fabf4c66fa2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 17:54:00 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33998
I20260929 17:54:00.251044 10785 tablet_service.cc:1514] Processing DeleteTablet for tablet a36815a14c294d7888a69fabf4c66fa2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 17:54:00 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33080
I20260929 17:54:00.251497 11159 tablet_replica.cc:330] T a36815a14c294d7888a69fabf4c66fa2 P 68617a4bfcea4a818f4856ac453a3cd4: stopping tablet replica
I20260929 17:54:00.251889 11159 raft_consensus.cc:2229] T a36815a14c294d7888a69fabf4c66fa2 P 68617a4bfcea4a818f4856ac453a3cd4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 17:54:00.252009 11053 tablet_service.cc:1514] Processing DeleteTablet for tablet a36815a14c294d7888a69fabf4c66fa2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 17:54:00 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48602
I20260929 17:54:00.252310 11159 raft_consensus.cc:2258] T a36815a14c294d7888a69fabf4c66fa2 P 68617a4bfcea4a818f4856ac453a3cd4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 17:54:00.253134 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskup3Soe/build/debug/bin/kudu with pid 10722
I20260929 17:54:00.257593 11160 tablet_replica.cc:330] T a36815a14c294d7888a69fabf4c66fa2 P de2522023d1242cbbf12ba4fb91e5549: stopping tablet replica
I20260929 17:54:00.257961 11160 raft_consensus.cc:2229] T a36815a14c294d7888a69fabf4c66fa2 P de2522023d1242cbbf12ba4fb91e5549 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 17:54:00.258291 11160 raft_consensus.cc:2258] T a36815a14c294d7888a69fabf4c66fa2 P de2522023d1242cbbf12ba4fb91e5549 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 17:54:00.258642 11158 tablet_replica.cc:330] T a36815a14c294d7888a69fabf4c66fa2 P 5f51e69ac560469098e414a536ede5a0: stopping tablet replica
I20260929 17:54:00.259173 11158 raft_consensus.cc:2229] T a36815a14c294d7888a69fabf4c66fa2 P 5f51e69ac560469098e414a536ede5a0 [term 1 LEADER]: Raft consensus shutting down.
W20260929 17:54:00.259598 10645 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:45175 (error 108)
I20260929 17:54:00.259621 11158 raft_consensus.cc:2258] T a36815a14c294d7888a69fabf4c66fa2 P 5f51e69ac560469098e414a536ede5a0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 17:54:00.260458 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskup3Soe/build/debug/bin/kudu with pid 10856
I20260929 17:54:00.260499 11160 ts_tablet_manager.cc:1905] T a36815a14c294d7888a69fabf4c66fa2 P de2522023d1242cbbf12ba4fb91e5549: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 17:54:00.262251 11131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:54:00.261944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45175 (local address 127.0.0.1:33082)
0929 17:54:00.262086 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0929 17:54:00.262161 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260929 17:54:00.262436 10645 catalog_manager.cc:4584] TS 68617a4bfcea4a818f4856ac453a3cd4 (127.0.98.1:45175): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a36815a14c294d7888a69fabf4c66fa2: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45175: connect: Connection refused (error 111)
I20260929 17:54:00.264135 11160 ts_tablet_manager.cc:1918] T a36815a14c294d7888a69fabf4c66fa2 P de2522023d1242cbbf12ba4fb91e5549: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 17:54:00.264288 11160 log.cc:1196] T a36815a14c294d7888a69fabf4c66fa2 P de2522023d1242cbbf12ba4fb91e5549: Deleting WAL directory at /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790704375975685-392-0/minicluster-data/ts-2/wal/wals/a36815a14c294d7888a69fabf4c66fa2
I20260929 17:54:00.264557 11160 ts_tablet_manager.cc:1939] T a36815a14c294d7888a69fabf4c66fa2 P de2522023d1242cbbf12ba4fb91e5549: Deleting consensus metadata
I20260929 17:54:00.265776 10644 catalog_manager.cc:4841] TS de2522023d1242cbbf12ba4fb91e5549 (127.0.98.3:46615): tablet a36815a14c294d7888a69fabf4c66fa2 (table test-table [id=e724fc0a726641c1b0e57f583d0af58b]) successfully deleted
I20260929 17:54:00.266307 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskup3Soe/build/debug/bin/kudu with pid 10990
I20260929 17:54:00.266551 11131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:54:00.266280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45499 (local address 127.0.0.1:34010)
0929 17:54:00.266428 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0929 17:54:00.266478 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:45499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260929 17:54:00.270959 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskup3Soe/build/debug/bin/kudu with pid 10628
2026-09-29T17:54:00Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2263 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704375975685-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 17:54:00 dist-test-slave-53nh krb5kdc[11164](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 17:54:00 dist-test-slave-53nh krb5kdc[11164](info): set up 2 sockets
Sep 29 17:54:00 dist-test-slave-53nh krb5kdc[11164](info): commencing operation
krb5kdc: starting...
W20260929 17:54:01.225960 392 mini_kdc.cc:121] Time spent starting KDC: real 0.935s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704375975685-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704375975685-392-0/krb5kdc/test-user.keytab.
Sep 29 17:54:01 dist-test-slave-53nh krb5kdc[11164](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704441, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T17:54:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T17:54:01Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704375975685-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704375975685-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704375975685-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704375975685-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704375975685-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704375975685-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260929 17:54:01.387936 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704375975685-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskup3Soe/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704375975685-392-0/minicluster-data/postgres -l logfile start
2026-09-29 17:54:03.802 UTC [11194] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-29 17:54:03.802 UTC [11194] LOG: listening on IPv4 address "127.0.98.20", port 33185
2026-09-29 17:54:03.805 UTC [11194] LOG: listening on Unix socket "/tmp/.s.PGSQL.33185"
2026-09-29 17:54:03.825 UTC [11197] LOG: database system was shut down at 2026-09-29 17:54:02 UTC
2026-09-29 17:54:03.829 UTC [11194] LOG: database system is ready to accept connections
I20260929 17:54:04.701377 392 mini_postgres.cc:96] Postgres bound to 33185
2026-09-29 17:54:04.707 UTC [11205] FATAL: database "slave" does not exist
127.0.98.20:33185 - accepting connections
I20260929 17:54:04.707674 392 mini_ranger.cc:161] Starting Ranger...
I20260929 17:54:04.722136 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260929 17:54:05.172253 392 mini_ranger.cc:87] Created ranger Postgres database
I20260929 17:54:05.172333 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704375975685-392-0/minicluster-data/ranger-admin
2026-09-29 17:54:05,458 [I] DB FLAVOR :POSTGRES
2026-09-29 17:54:05,458 [I] --------- Verifying Ranger DB connection ---------
2026-09-29 17:54:05,458 [I] Checking connection..
2026-09-29 17:54:05,458 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskup3Soe/build/debug/bin/postgresql.jar:/tmp/dist-test-taskup3Soe/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:33185/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-29 17:54:05,715 [I] Checking connection passed.
2026-09-29 17:54:05,715 [I] --------- Verifying version history table ---------
2026-09-29 17:54:05,715 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskup3Soe/build/debug/bin/postgresql.jar:/tmp/dist-test-taskup3Soe/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:33185/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 17:54:05,951 [I] Table x_db_version_h does not exist in database ranger
2026-09-29 17:54:05,951 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskup3Soe/build/debug/bin/postgresql.jar:/tmp/dist-test-taskup3Soe/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:33185/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 17:54:06,172 [I] Table x_db_version_h does not exist in database ranger
2026-09-29 17:54:06,173 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-29 17:54:06,173 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskup3Soe/build/debug/bin/postgresql.jar:/tmp/dist-test-taskup3Soe/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:33185/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskup3Soe/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-29 17:54:06.391 UTC [11269] WARNING: there is no transaction in progress
2026-09-29 17:54:06,397 [I] create_dbversion_catalog.sql file imported successfully
2026-09-29 17:54:06,397 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskup3Soe/build/debug/bin/postgresql.jar:/tmp/dist-test-taskup3Soe/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:33185/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 17:54:06,630 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-29 17:54:06,630 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-29 17:54:06,630 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskup3Soe/build/debug/bin/postgresql.jar:/tmp/dist-test-taskup3Soe/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:33185/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-29 17:54:06,881 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskup3Soe/build/debug/bin/postgresql.jar:/tmp/dist-test-taskup3Soe/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:33185/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-29 17:54:07,109 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskup3Soe/build/debug/bin/postgresql.jar:/tmp/dist-test-taskup3Soe/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:33185/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-53nh.c.gcp-upstream.internal','N') ;"
2026-09-29 17:54:07,331 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskup3Soe/build/debug/bin/postgresql.jar:/tmp/dist-test-taskup3Soe/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:33185/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-29 17:54:07,583 [I] Table x_portal_user does not exist in database ranger
2026-09-29 17:54:07,583 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskup3Soe/build/debug/bin/postgresql.jar:/tmp/dist-test-taskup3Soe/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:33185/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-29 17:54:07,815 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-29 17:54:07,816 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskup3Soe/build/debug/bin/postgresql.jar:/tmp/dist-test-taskup3Soe/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:33185/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-29 17:54:08,037 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskup3Soe/build/debug/bin/postgresql.jar:/tmp/dist-test-taskup3Soe/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:33185/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-29 17:54:08,287 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-29 17:54:08,287 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskup3Soe/build/debug/bin/postgresql.jar:/tmp/dist-test-taskup3Soe/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:33185/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskup3Soe/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-29 17:54:09.245 UTC [11377] WARNING: there is no transaction in progress
2026-09-29 17:54:09.262 UTC [11377] WARNING: there is no transaction in progress
2026-09-29 17:54:09.277 UTC [11377] WARNING: there is no transaction in progress
2026-09-29 17:54:09.292 UTC [11377] WARNING: there is no transaction in progress
2026-09-29 17:54:09.307 UTC [11377] WARNING: there is no transaction in progress
2026-09-29 17:54:09.325 UTC [11377] WARNING: there is no transaction in progress
2026-09-29 17:54:09.378 UTC [11377] WARNING: there is no transaction in progress
2026-09-29 17:54:09.384 UTC [11377] WARNING: there is no transaction in progress
2026-09-29 17:54:09.398 UTC [11377] WARNING: there is no transaction in progress
2026-09-29 17:54:09.408 UTC [11377] WARNING: there is no transaction in progress
2026-09-29 17:54:09.419 UTC [11377] WARNING: there is no transaction in progress
2026-09-29 17:54:09.432 UTC [11377] WARNING: there is no transaction in progress
2026-09-29 17:54:09.442 UTC [11377] WARNING: there is no transaction in progress
2026-09-29 17:54:09.448 UTC [11377] WARNING: there is no transaction in progress
2026-09-29 17:54:09.454 UTC [11377] WARNING: there is no transaction in progress
2026-09-29 17:54:09.460 UTC [11377] WARNING: there is no transaction in progress
2026-09-29 17:54:10,068 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-29 17:54:10,068 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskup3Soe/build/debug/bin/postgresql.jar:/tmp/dist-test-taskup3Soe/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:33185/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-29 17:54:10,315 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-29 17:54:10,315 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskup3Soe/build/debug/bin/postgresql.jar:/tmp/dist-test-taskup3Soe/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:33185/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-29 17:54:10,566 [I] Table x_portal_user already exists in database 'ranger'
2026-09-29 17:54:10,567 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskup3Soe/build/debug/bin/postgresql.jar:/tmp/dist-test-taskup3Soe/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:33185/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-29 17:54:10,802 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-29 17:54:10,802 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskup3Soe/build/debug/bin/postgresql.jar:/tmp/dist-test-taskup3Soe/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:33185/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-29 17:54:11,046 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskup3Soe/build/debug/bin/postgresql.jar:/tmp/dist-test-taskup3Soe/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:33185/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-29 17:54:11,281 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskup3Soe/build/debug/bin/postgresql.jar:/tmp/dist-test-taskup3Soe/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:33185/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-53nh.c.gcp-upstream.internal';"
2026-09-29 17:54:11,518 [I] CORE_DB_SCHEMA import status has been updated
2026-09-29 17:54:11,518 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskup3Soe/build/debug/bin/postgresql.jar:/tmp/dist-test-taskup3Soe/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:33185/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-29 17:54:11,761 [I] DB_PATCHES have already been applied
I20260929 17:54:11.766381 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704375975685-392-0/minicluster-data/ranger-admin:/tmp/dist-test-taskup3Soe/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskup3Soe/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskup3Soe/build/debug/bin/postgresql.jar:/tmp/dist-test-taskup3Soe/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260929 17:54:11.766444 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260929 17:54:11.769572 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:59651
Sep 29, 2026 5:54:12 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 29, 2026 5:54:12 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 29, 2026 5:54:12 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704375975685-392-0/minicluster-data/ranger-admin/ews/webapp
Sep 29, 2026 5:54:12 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704375975685-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 29, 2026 5:54:12 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704375975685-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 29, 2026 5:54:12 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704375975685-392-0/minicluster-data/ranger-admin/ews/webapp].
Sep 29, 2026 5:54:12 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260929 17:54:12.767465 392 mini_ranger.cc:160] Time spent starting Ranger: real 8.060s user 0.001s sys 0.008s
Sep 29, 2026 5:54:31 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 29, 2026 5:54:31 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Sep 29, 2026 5:54:31 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 29, 2026 5:54:31 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 29, 2026 5:54:31 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 29, 2026 5:54:31 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 29, 2026 5:54:31 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 29, 2026 5:54:31 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 29, 2026 5:54:31 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 29, 2026 5:54:31 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 29, 2026 5:54:31 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 29, 2026 5:54:31 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 29, 2026 5:54:31 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 29, 2026 5:54:31 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 29, 2026 5:54:31 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 29, 2026 5:54:31 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 29, 2026 5:54:31 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 29, 2026 5:54:32 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260929 17:54:33.099399 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704375975685-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704375975685-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704375975685-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704375975685-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 29 17:54:33 dist-test-slave-53nh krb5kdc[11164](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704473, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260929 17:54:33.198489 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260929 17:54:33.208340 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260929 17:54:34.086877 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260929 17:54:34.086970 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704375975685-392-0/minicluster-data/ranger-kms
I20260929 17:54:34.087069 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:33185
2026-09-29 17:54:34,256 [I] DB FLAVOR :POSTGRES
2026-09-29 17:54:34,256 [I] --------- Verifying Ranger DB connection ---------
2026-09-29 17:54:34,256 [I] Checking connection
2026-09-29 17:54:34,256 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskup3Soe/build/debug/bin/postgresql.jar:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704375975685-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:33185/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-29 17:54:34,515 [I] connection success
2026-09-29 17:54:34,515 [I] --------- Verifying Ranger DB tables ---------
2026-09-29 17:54:34,515 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-29 17:54:34,515 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskup3Soe/build/debug/bin/postgresql.jar:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704375975685-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:33185/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-29 17:54:34,741 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-29 17:54:34,742 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-29 17:54:34,742 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-29 17:54:34,742 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskup3Soe/build/debug/bin/postgresql.jar:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704375975685-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:33185/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704375975685-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-29 17:54:34,998 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260929 17:54:35.486270 392 mini_ranger_kms.cc:310] Created kms service
I20260929 17:54:35.657178 392 mini_ranger_kms.cc:326] Created kudu user
I20260929 17:54:35.697067 392 mini_ranger_kms.cc:343] Created rangerkms user
Sep 29 17:54:35 dist-test-slave-53nh krb5kdc[11164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704473, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260929 17:54:36.112990 392 mini_ranger_kms.cc:384] Added ranger policy
I20260929 17:54:36.113202 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704375975685-392-0/minicluster-data/ranger-kms:/tmp/dist-test-taskup3Soe/build/debug/bin/postgresql.jar:/tmp/dist-test-taskup3Soe/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskup3Soe/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260929 17:54:36.113234 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260929 17:54:36.116672 392 mini_ranger_kms.cc:276] Ranger KMS PID: 11557
I20260929 17:54:36.116806 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:48739
Sep 29, 2026 5:54:37 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 29, 2026 5:54:37 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 29, 2026 5:54:37 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704375975685-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 29, 2026 5:54:37 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704375975685-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 29, 2026 5:54:37 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704375975685-392-0/minicluster-data/ranger-kms/ews/webapp].
Sep 29, 2026 5:54:37 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Sep 29 17:54:38 dist-test-slave-53nh krb5kdc[11164](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790704478, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:54:38.129756 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 4.931s user 0.004s sys 0.149s
I20260929 17:54:38.129894 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260929 17:54:38.130023 392 mini_ranger_kms.cc:401] 127.0.98.20:48739/kms/v1/keys
Sep 29 17:54:38 dist-test-slave-53nh krb5kdc[11164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704478, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 29 17:54:38 dist-test-slave-53nh krb5kdc[11164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704478, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 29 17:54:41 dist-test-slave-53nh krb5kdc[11164](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704481, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704375975685-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704375975685-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 17:54:41.257947 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskup3Soe/build/debug/bin/kudu
/tmp/dist-test-taskup3Soe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704375975685-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704375975685-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704375975685-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704375975685-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39561
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704375975685-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39561
--ranger_config_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704375975685-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:48739/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704375975685-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704375975685-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704375975685-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:54:41.389181 11608 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:54:41.389457 11608 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:54:41.389514 11608 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:54:41.393561 11608 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 17:54:41.393639 11608 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:54:41.393664 11608 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:54:41.393687 11608 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 17:54:41.393702 11608 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 17:54:41.399331 11608 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704375975685-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704375975685-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:48739/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39561
--ranger_config_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704375975685-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704375975685-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39561
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704375975685-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11608
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704375975685-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:54:41.400388 11608 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:54:41.401439 11608 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:54:41.409960 11614 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:54:41.409941 11616 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:54:41.409961 11613 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:54:41.410984 11608 server_base.cc:1034] running on GCE node
I20260929 17:54:41.411665 11608 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:54:41.413147 11608 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:54:41.414348 11608 hybrid_clock.cc:648] HybridClock initialized: now 1790704481414314 us; error 59 us; skew 500 ppm
Sep 29 17:54:41 dist-test-slave-53nh krb5kdc[11164](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704481, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:54:41.424670 11608 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 17:54:41.425890 11608 webserver.cc:460] Webserver started at http://127.0.98.62:45721/ using document root <none> and password file <none>
I20260929 17:54:41.426570 11608 fs_manager.cc:359] Metadata directory not provided
I20260929 17:54:41.426643 11608 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:54:41.426857 11608 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 17:54:41 dist-test-slave-53nh krb5kdc[11164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704481, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260929 17:54:41.472962 11608 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704375975685-392-0/minicluster-data/master-0/data/instance:
uuid: "50cc925367d547a2941d686dcfe17a63"
format_stamp: "Formatted at 2026-09-29 17:54:41 on dist-test-slave-53nh"
server_key: "0b6929e2b0e2f08f4e7a9319260561a4"
server_key_iv: "71ec82c752c1f2374ea3b4956f9f755b"
server_key_version: "kuduclusterkey@0"
I20260929 17:54:41.473639 11608 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704375975685-392-0/minicluster-data/master-0/wal/instance:
uuid: "50cc925367d547a2941d686dcfe17a63"
format_stamp: "Formatted at 2026-09-29 17:54:41 on dist-test-slave-53nh"
server_key: "0b6929e2b0e2f08f4e7a9319260561a4"
server_key_iv: "71ec82c752c1f2374ea3b4956f9f755b"
server_key_version: "kuduclusterkey@0"
I20260929 17:54:41.477595 11608 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260929 17:54:41.494675 11623 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:54:41.496291 11608 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 17:54:41.496435 11608 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704375975685-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704375975685-392-0/minicluster-data/master-0/wal
uuid: "50cc925367d547a2941d686dcfe17a63"
format_stamp: "Formatted at 2026-09-29 17:54:41 on dist-test-slave-53nh"
server_key: "0b6929e2b0e2f08f4e7a9319260561a4"
server_key_iv: "71ec82c752c1f2374ea3b4956f9f755b"
server_key_version: "kuduclusterkey@0"
I20260929 17:54:41.496573 11608 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704375975685-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704375975685-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704375975685-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:54:41.512143 11608 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:54:41.515131 11608 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:54:41.515319 11608 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:54:41.523728 11608 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39561
I20260929 17:54:41.523726 11675 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39561 every 8 connection(s)
I20260929 17:54:41.524833 11608 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704375975685-392-0/minicluster-data/master-0/data/info.pb
I20260929 17:54:41.525799 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskup3Soe/build/debug/bin/kudu as pid 11608
I20260929 17:54:41.525907 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704375975685-392-0/minicluster-data/master-0/wal/instance
I20260929 17:54:41.526186 392 external_mini_cluster.cc:1428] Setting key 214303c89ac8daa56450b9330c2f4b8e
I20260929 17:54:41.528156 11676 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:54:41.534011 11676 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50cc925367d547a2941d686dcfe17a63: Bootstrap starting.
I20260929 17:54:41.536523 11676 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 50cc925367d547a2941d686dcfe17a63: Neither blocks nor log segments found. Creating new log.
Sep 29 17:54:41 dist-test-slave-53nh krb5kdc[11164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704481, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 17:54:41.537307 11676 log.cc:824] T 00000000000000000000000000000000 P 50cc925367d547a2941d686dcfe17a63: Log is configured to *not* fsync() on all Append() calls
I20260929 17:54:41.539471 11676 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50cc925367d547a2941d686dcfe17a63: No bootstrap required, opened a new log
I20260929 17:54:41.542156 11679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:54:41.527510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42136 (local address 127.0.98.62:39561)
0929 17:54:41.527980 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0929 17:54:41.527997 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 17:54:41.528181 (+ 184us) server_negotiation.cc:407] Connection header received
0929 17:54:41.529161 (+ 980us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:54:41.529184 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:54:41.529588 (+ 404us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:54:41.529927 (+ 339us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:54:41.530734 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:54:41.532258 (+ 1524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:54:41.533455 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:54:41.534168 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:54:41.534325 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:54:41.537269 (+ 2944us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:54:41.537275 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:54:41.537290 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:54:41.537321 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:54:41.539465 (+ 2144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:54:41.539907 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:54:41.539913 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:54:41.539919 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:54:41.540025 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:54:41.540277 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:54:41.540280 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:54:41.540282 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:54:41.540655 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:54:41.540825 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0929 17:54:41.541140 (+ 315us) server_negotiation.cc:299] Negotiation successful
0929 17:54:41.541470 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":144,"threads_started":1}
I20260929 17:54:41.543907 11676 raft_consensus.cc:348] T 00000000000000000000000000000000 P 50cc925367d547a2941d686dcfe17a63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50cc925367d547a2941d686dcfe17a63" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39561 } }
I20260929 17:54:41.544270 11676 raft_consensus.cc:374] T 00000000000000000000000000000000 P 50cc925367d547a2941d686dcfe17a63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:54:41.544394 11676 raft_consensus.cc:729] T 00000000000000000000000000000000 P 50cc925367d547a2941d686dcfe17a63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50cc925367d547a2941d686dcfe17a63, State: Initialized, Role: FOLLOWER
I20260929 17:54:41.544853 11676 consensus_queue.cc:260] T 00000000000000000000000000000000 P 50cc925367d547a2941d686dcfe17a63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50cc925367d547a2941d686dcfe17a63" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39561 } }
I20260929 17:54:41.545032 11676 raft_consensus.cc:388] T 00000000000000000000000000000000 P 50cc925367d547a2941d686dcfe17a63 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 17:54:41.545092 11676 raft_consensus.cc:482] T 00000000000000000000000000000000 P 50cc925367d547a2941d686dcfe17a63 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 17:54:41.545181 11676 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 50cc925367d547a2941d686dcfe17a63 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:54:41.546232 11676 raft_consensus.cc:504] T 00000000000000000000000000000000 P 50cc925367d547a2941d686dcfe17a63 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50cc925367d547a2941d686dcfe17a63" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39561 } }
I20260929 17:54:41.546578 11676 leader_election.cc:302] T 00000000000000000000000000000000 P 50cc925367d547a2941d686dcfe17a63 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 50cc925367d547a2941d686dcfe17a63; no voters:
I20260929 17:54:41.546969 11676 leader_election.cc:288] T 00000000000000000000000000000000 P 50cc925367d547a2941d686dcfe17a63 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 17:54:41.547030 11681 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 50cc925367d547a2941d686dcfe17a63 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 17:54:41.547260 11681 raft_consensus.cc:686] T 00000000000000000000000000000000 P 50cc925367d547a2941d686dcfe17a63 [term 1 LEADER]: Becoming Leader. State: Replica: 50cc925367d547a2941d686dcfe17a63, State: Running, Role: LEADER
I20260929 17:54:41.547889 11681 consensus_queue.cc:237] T 00000000000000000000000000000000 P 50cc925367d547a2941d686dcfe17a63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50cc925367d547a2941d686dcfe17a63" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39561 } }
I20260929 17:54:41.548262 11676 sys_catalog.cc:564] T 00000000000000000000000000000000 P 50cc925367d547a2941d686dcfe17a63 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 17:54:41.549499 11682 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50cc925367d547a2941d686dcfe17a63 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "50cc925367d547a2941d686dcfe17a63" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50cc925367d547a2941d686dcfe17a63" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39561 } } }
I20260929 17:54:41.549639 11682 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50cc925367d547a2941d686dcfe17a63 [sys.catalog]: This master's current role is: LEADER
I20260929 17:54:41.549924 11683 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50cc925367d547a2941d686dcfe17a63 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 50cc925367d547a2941d686dcfe17a63. Latest consensus state: current_term: 1 leader_uuid: "50cc925367d547a2941d686dcfe17a63" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50cc925367d547a2941d686dcfe17a63" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39561 } } }
I20260929 17:54:41.550061 11683 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50cc925367d547a2941d686dcfe17a63 [sys.catalog]: This master's current role is: LEADER
I20260929 17:54:41.550446 11676 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704375975685-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260929 17:54:41.551276 11690 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 17:54:41.557039 11690 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 17:54:41.564493 11690 catalog_manager.cc:1347] Generated new cluster ID: 1bcb306c08174b85acd729f6011ff0cf
I20260929 17:54:41.564555 11690 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 17:54:41.575366 11690 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 17:54:41.576728 11690 catalog_manager.cc:1504] Loading token signing keys...
I20260929 17:54:41.584292 11690 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 50cc925367d547a2941d686dcfe17a63: Generated new TSK 0
I20260929 17:54:41.585268 11690 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704375975685-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704375975685-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260929 17:54:42.311512 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskup3Soe/build/debug/bin/kudu
/tmp/dist-test-taskup3Soe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704375975685-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704375975685-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704375975685-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704375975685-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:48739/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39561
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704375975685-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704375975685-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704375975685-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704375975685-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:54:42.454340 11721 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:54:42.454591 11721 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:54:42.454632 11721 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:54:42.459105 11721 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:54:42.459167 11721 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:54:42.459226 11721 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 17:54:42.465817 11721 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704375975685-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704375975685-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:48739/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704375975685-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704375975685-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39561
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11721
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704375975685-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:54:42.467036 11721 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:54:42.468242 11721 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:54:42.477610 11729 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:54:42.477998 11721 server_base.cc:1034] running on GCE node
W20260929 17:54:42.477995 11731 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:54:42.478276 11728 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:54:42.478698 11721 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:54:42.479328 11721 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:54:42.482599 11721 hybrid_clock.cc:648] HybridClock initialized: now 1790704482482570 us; error 36 us; skew 500 ppm
Sep 29 17:54:42 dist-test-slave-53nh krb5kdc[11164](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704482, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:54:42.485333 11721 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260929 17:54:42.486207 11721 webserver.cc:460] Webserver started at http://127.0.98.1:34389/ using document root <none> and password file <none>
I20260929 17:54:42.486737 11721 fs_manager.cc:359] Metadata directory not provided
I20260929 17:54:42.486801 11721 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:54:42.486984 11721 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 17:54:42 dist-test-slave-53nh krb5kdc[11164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704482, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260929 17:54:42.504460 11721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704375975685-392-0/minicluster-data/ts-0/data/instance:
uuid: "f0ee61e1dba647f2961427e34e3c54b3"
format_stamp: "Formatted at 2026-09-29 17:54:42 on dist-test-slave-53nh"
server_key: "84e584418ae449a3b8bb42cf02f27cd4"
server_key_iv: "5c03fc9881e7f5ea1bfaaa1ad85f98b4"
server_key_version: "kuduclusterkey@0"
I20260929 17:54:42.505513 11721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704375975685-392-0/minicluster-data/ts-0/wal/instance:
uuid: "f0ee61e1dba647f2961427e34e3c54b3"
format_stamp: "Formatted at 2026-09-29 17:54:42 on dist-test-slave-53nh"
server_key: "84e584418ae449a3b8bb42cf02f27cd4"
server_key_iv: "5c03fc9881e7f5ea1bfaaa1ad85f98b4"
server_key_version: "kuduclusterkey@0"
I20260929 17:54:42.513042 11721 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.004s sys 0.000s
I20260929 17:54:42.548458 11738 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:54:42.550468 11721 fs_manager.cc:725] Time spent opening block manager: real 0.023s user 0.003s sys 0.001s
I20260929 17:54:42.550617 11721 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704375975685-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704375975685-392-0/minicluster-data/ts-0/wal
uuid: "f0ee61e1dba647f2961427e34e3c54b3"
format_stamp: "Formatted at 2026-09-29 17:54:42 on dist-test-slave-53nh"
server_key: "84e584418ae449a3b8bb42cf02f27cd4"
server_key_iv: "5c03fc9881e7f5ea1bfaaa1ad85f98b4"
server_key_version: "kuduclusterkey@0"
I20260929 17:54:42.550758 11721 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704375975685-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704375975685-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704375975685-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:54:42.581643 11721 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:54:42.585314 11721 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:54:42.585572 11721 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:54:42.586259 11721 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 17:54:42.587283 11721 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:54:42.587380 11721 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:54:42.587503 11721 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:54:42.587607 11721 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 17:54:42.600711 11721 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34395
I20260929 17:54:42.601934 11721 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704375975685-392-0/minicluster-data/ts-0/data/info.pb
I20260929 17:54:42.603971 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskup3Soe/build/debug/bin/kudu as pid 11721
I20260929 17:54:42.604089 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704375975685-392-0/minicluster-data/ts-0/wal/instance
I20260929 17:54:42.604373 392 external_mini_cluster.cc:1428] Setting key aecfae6ba0ce6389929168e528d856fe
I20260929 17:54:42.614872 11853 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34395 every 8 connection(s)
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Sep 29 17:54:42 dist-test-slave-53nh krb5kdc[11164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704482, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 17:54:42.638952 11858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:54:42.616585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52085 (local address 127.0.98.62:39561)
0929 17:54:42.624149 (+ 7564us) server_negotiation.cc:206] Beginning negotiation
0929 17:54:42.624154 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:54:42.624169 (+ 15us) server_negotiation.cc:407] Connection header received
0929 17:54:42.624214 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:54:42.624216 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:54:42.624265 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:54:42.624354 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:54:42.626301 (+ 1947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:54:42.627179 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:54:42.631223 (+ 4044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:54:42.631608 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:54:42.631690 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:54:42.634747 (+ 3057us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:54:42.634751 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:54:42.634753 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:54:42.634782 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:54:42.636579 (+ 1797us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:54:42.637184 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:54:42.637188 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:54:42.637190 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:54:42.637274 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:54:42.637658 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:54:42.637661 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:54:42.637663 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:54:42.637880 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:54:42.637963 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0929 17:54:42.638640 (+ 677us) server_negotiation.cc:299] Negotiation successful
0929 17:54:42.638766 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7472,"thread_start_us":82,"threads_started":1}
I20260929 17:54:42.641057 11857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:54:42.615414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39561 (local address 127.0.98.1:52085)
0929 17:54:42.622430 (+ 7016us) negotiation.cc:107] Waiting for socket to connect
0929 17:54:42.622509 (+ 79us) client_negotiation.cc:174] Beginning negotiation
0929 17:54:42.623675 (+ 1166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:54:42.625101 (+ 1426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:54:42.625112 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:54:42.625442 (+ 330us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:54:42.626125 (+ 683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:54:42.626144 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:54:42.627525 (+ 1381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:54:42.627529 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:54:42.631073 (+ 3544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:54:42.631085 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:54:42.631913 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:54:42.631916 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:54:42.632131 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:54:42.632835 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:54:42.632856 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:54:42.634538 (+ 1682us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:54:42.636711 (+ 2173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:54:42.636717 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:54:42.636734 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:54:42.637075 (+ 341us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:54:42.637370 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:54:42.637373 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:54:42.637375 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:54:42.637538 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:54:42.638027 (+ 489us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:54:42.638034 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:54:42.638351 (+ 317us) client_negotiation.cc:769] Sending connection context
0929 17:54:42.639624 (+ 1273us) client_negotiation.cc:240] Negotiation successful
0929 17:54:42.639953 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6729,"thread_start_us":167,"threads_started":1}
I20260929 17:54:42.642831 11854 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39561
I20260929 17:54:42.643091 11854 heartbeater.cc:464] Registering TS with master...
I20260929 17:54:42.643692 11854 heartbeater.cc:511] Master 127.0.98.62:39561 requested a full tablet report, sending...
I20260929 17:54:42.645505 11640 ts_manager.cc:194] Registered new tserver with Master: f0ee61e1dba647f2961427e34e3c54b3 (127.0.98.1:34395)
I20260929 17:54:42.647006 11640 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:52085
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704375975685-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704375975685-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260929 17:54:42.669045 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskup3Soe/build/debug/bin/kudu
/tmp/dist-test-taskup3Soe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704375975685-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704375975685-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704375975685-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704375975685-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:48739/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39561
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704375975685-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704375975685-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704375975685-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704375975685-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 17:54:42 dist-test-slave-53nh krb5kdc[11164](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790704482, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260929 17:54:42.831664 11860 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:54:42.831950 11860 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:54:42.832010 11860 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:54:42.836796 11860 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:54:42.836925 11860 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:54:42.837064 11860 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 17:54:42.842796 11860 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704375975685-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704375975685-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:48739/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704375975685-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704375975685-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39561
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11860
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704375975685-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:54:42.843844 11860 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:54:42.844875 11860 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:54:42.855583 11866 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:54:42.855845 11860 server_base.cc:1034] running on GCE node
W20260929 17:54:42.855535 11869 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:54:42.857141 11867 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:54:42.857555 11860 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:54:42.858235 11860 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:54:42.865598 11860 hybrid_clock.cc:648] HybridClock initialized: now 1790704482865511 us; error 102 us; skew 500 ppm
Sep 29 17:54:42 dist-test-slave-53nh krb5kdc[11164](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704482, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:54:42.869347 11860 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260929 17:54:42.870457 11860 webserver.cc:460] Webserver started at http://127.0.98.2:41903/ using document root <none> and password file <none>
I20260929 17:54:42.871184 11860 fs_manager.cc:359] Metadata directory not provided
I20260929 17:54:42.871287 11860 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:54:42.871523 11860 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 17:54:42 dist-test-slave-53nh krb5kdc[11164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704482, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260929 17:54:42.887059 11860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704375975685-392-0/minicluster-data/ts-1/data/instance:
uuid: "91ae4db6b5674cfe8673df38b233fe1e"
format_stamp: "Formatted at 2026-09-29 17:54:42 on dist-test-slave-53nh"
server_key: "3759a5f006b3de97dcfe530dc51aeea9"
server_key_iv: "3374dd93ed08274ba2af073f326647dd"
server_key_version: "kuduclusterkey@0"
I20260929 17:54:42.888065 11860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704375975685-392-0/minicluster-data/ts-1/wal/instance:
uuid: "91ae4db6b5674cfe8673df38b233fe1e"
format_stamp: "Formatted at 2026-09-29 17:54:42 on dist-test-slave-53nh"
server_key: "3759a5f006b3de97dcfe530dc51aeea9"
server_key_iv: "3374dd93ed08274ba2af073f326647dd"
server_key_version: "kuduclusterkey@0"
I20260929 17:54:42.893154 11860 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 17:54:42.913288 11876 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:54:42.915081 11860 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.004s sys 0.000s
I20260929 17:54:42.915287 11860 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704375975685-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704375975685-392-0/minicluster-data/ts-1/wal
uuid: "91ae4db6b5674cfe8673df38b233fe1e"
format_stamp: "Formatted at 2026-09-29 17:54:42 on dist-test-slave-53nh"
server_key: "3759a5f006b3de97dcfe530dc51aeea9"
server_key_iv: "3374dd93ed08274ba2af073f326647dd"
server_key_version: "kuduclusterkey@0"
I20260929 17:54:42.915516 11860 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704375975685-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704375975685-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704375975685-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:54:42.940423 11860 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:54:42.945405 11860 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:54:42.945678 11860 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:54:42.946475 11860 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 17:54:42.947600 11860 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:54:42.947710 11860 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:54:42.947810 11860 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:54:42.947873 11860 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 17:54:42.962543 11860 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40357
I20260929 17:54:42.963836 11860 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704375975685-392-0/minicluster-data/ts-1/data/info.pb
I20260929 17:54:42.965575 11989 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40357 every 8 connection(s)
I20260929 17:54:42.972880 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskup3Soe/build/debug/bin/kudu as pid 11860
I20260929 17:54:42.973030 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704375975685-392-0/minicluster-data/ts-1/wal/instance
I20260929 17:54:42.973340 392 external_mini_cluster.cc:1428] Setting key 1d738fda2c99f4bdf6d47927ef30c483
Sep 29 17:54:42 dist-test-slave-53nh krb5kdc[11164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704482, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260929 17:54:42.988266 11858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:54:42.967751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:32929 (local address 127.0.98.62:39561)
0929 17:54:42.967897 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0929 17:54:42.967902 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:54:42.971590 (+ 3688us) server_negotiation.cc:407] Connection header received
0929 17:54:42.973630 (+ 2040us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:54:42.973634 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:54:42.973695 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:54:42.973752 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:54:42.975517 (+ 1765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:54:42.976352 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:54:42.978123 (+ 1771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:54:42.978483 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:54:42.978556 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:54:42.983073 (+ 4517us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:54:42.983078 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:54:42.983080 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:54:42.983112 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:54:42.984934 (+ 1822us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:54:42.986417 (+ 1483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:54:42.986421 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:54:42.986423 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:54:42.986515 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:54:42.986917 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:54:42.986920 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:54:42.986922 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:54:42.987144 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:54:42.987220 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0929 17:54:42.987962 (+ 742us) server_negotiation.cc:299] Negotiation successful
0929 17:54:42.988103 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260929 17:54:42.989297 11992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:54:42.968252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39561 (local address 127.0.98.2:32929)
0929 17:54:42.971386 (+ 3134us) negotiation.cc:107] Waiting for socket to connect
0929 17:54:42.971451 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0929 17:54:42.973298 (+ 1847us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:54:42.974244 (+ 946us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:54:42.974253 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:54:42.974620 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:54:42.975321 (+ 701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:54:42.975341 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:54:42.976525 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:54:42.976528 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:54:42.977987 (+ 1459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:54:42.977998 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:54:42.978761 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:54:42.978765 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:54:42.978971 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:54:42.981165 (+ 2194us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:54:42.981187 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:54:42.982878 (+ 1691us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:54:42.985149 (+ 2271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:54:42.985155 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:54:42.985174 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:54:42.985532 (+ 358us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:54:42.986624 (+ 1092us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:54:42.986627 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:54:42.986629 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:54:42.986811 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:54:42.987287 (+ 476us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:54:42.987294 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:54:42.987654 (+ 360us) client_negotiation.cc:769] Sending connection context
0929 17:54:42.987861 (+ 207us) client_negotiation.cc:240] Negotiation successful
0929 17:54:42.988207 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2891,"thread_start_us":152,"threads_started":1}
I20260929 17:54:42.990551 11990 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39561
I20260929 17:54:42.990809 11990 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260929 17:54:42.991415 11990 heartbeater.cc:511] Master 127.0.98.62:39561 requested a full tablet report, sending...
I20260929 17:54:42.992592 11640 ts_manager.cc:194] Registered new tserver with Master: 91ae4db6b5674cfe8673df38b233fe1e (127.0.98.2:40357)
I20260929 17:54:42.993337 11640 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:32929
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704375975685-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704375975685-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260929 17:54:43.016214 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskup3Soe/build/debug/bin/kudu
/tmp/dist-test-taskup3Soe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704375975685-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704375975685-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704375975685-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704375975685-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:48739/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39561
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704375975685-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704375975685-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704375975685-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704375975685-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:54:43.176636 11995 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:54:43.177361 11995 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:54:43.177487 11995 flags.cc:425] Enabled unsafe flag: --never_fsync=true
Sep 29 17:54:43 dist-test-slave-53nh krb5kdc[11164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704482, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
W20260929 17:54:43.184166 11995 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:54:43.184268 11995 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:54:43.184386 11995 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 17:54:43.192032 11995 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704375975685-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704375975685-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:48739/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704375975685-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704375975685-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39561
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11995
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704375975685-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:54:43.193548 11995 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:54:43.194844 11995 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:54:43.206893 12003 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:54:43.207357 12001 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:54:43.210315 11995 server_base.cc:1034] running on GCE node
W20260929 17:54:43.213325 12000 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:54:43.213887 11995 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:54:43.214730 11995 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:54:43.222136 11995 hybrid_clock.cc:648] HybridClock initialized: now 1790704483221887 us; error 339 us; skew 500 ppm
Sep 29 17:54:43 dist-test-slave-53nh krb5kdc[11164](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704483, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:54:43.225951 11995 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260929 17:54:43.227139 11995 webserver.cc:460] Webserver started at http://127.0.98.3:34449/ using document root <none> and password file <none>
I20260929 17:54:43.227900 11995 fs_manager.cc:359] Metadata directory not provided
I20260929 17:54:43.228030 11995 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:54:43.228302 11995 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 17:54:43 dist-test-slave-53nh krb5kdc[11164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704483, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260929 17:54:43.251710 11995 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704375975685-392-0/minicluster-data/ts-2/data/instance:
uuid: "576c9570349049bdae925f3e90b5b9e9"
format_stamp: "Formatted at 2026-09-29 17:54:43 on dist-test-slave-53nh"
server_key: "ff54228049f5f69548b4d353684f803c"
server_key_iv: "8b6f4507d965e2662f7b1a0cb2d1bd0a"
server_key_version: "kuduclusterkey@0"
I20260929 17:54:43.257479 11995 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704375975685-392-0/minicluster-data/ts-2/wal/instance:
uuid: "576c9570349049bdae925f3e90b5b9e9"
format_stamp: "Formatted at 2026-09-29 17:54:43 on dist-test-slave-53nh"
server_key: "ff54228049f5f69548b4d353684f803c"
server_key_iv: "8b6f4507d965e2662f7b1a0cb2d1bd0a"
server_key_version: "kuduclusterkey@0"
I20260929 17:54:43.272058 11995 fs_manager.cc:691] Time spent creating directory manager: real 0.014s user 0.004s sys 0.000s
I20260929 17:54:43.303375 12010 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:54:43.305128 11995 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260929 17:54:43.305326 11995 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704375975685-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704375975685-392-0/minicluster-data/ts-2/wal
uuid: "576c9570349049bdae925f3e90b5b9e9"
format_stamp: "Formatted at 2026-09-29 17:54:43 on dist-test-slave-53nh"
server_key: "ff54228049f5f69548b4d353684f803c"
server_key_iv: "8b6f4507d965e2662f7b1a0cb2d1bd0a"
server_key_version: "kuduclusterkey@0"
I20260929 17:54:43.305541 11995 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704375975685-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704375975685-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704375975685-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:54:43.322587 11995 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:54:43.326099 11995 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:54:43.326342 11995 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:54:43.327070 11995 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 17:54:43.328045 11995 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:54:43.328101 11995 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:54:43.328150 11995 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:54:43.328217 11995 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 17:54:43.338739 11995 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40765
I20260929 17:54:43.338809 12123 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40765 every 8 connection(s)
I20260929 17:54:43.339910 11995 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704375975685-392-0/minicluster-data/ts-2/data/info.pb
I20260929 17:54:43.349546 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskup3Soe/build/debug/bin/kudu as pid 11995
I20260929 17:54:43.349689 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704375975685-392-0/minicluster-data/ts-2/wal/instance
I20260929 17:54:43.349999 392 external_mini_cluster.cc:1428] Setting key d57e08aa63dfdcbf629ef9794265aa16
Sep 29 17:54:43 dist-test-slave-53nh krb5kdc[11164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704483, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 17:54:43.361790 11858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:54:43.341958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41149 (local address 127.0.98.62:39561)
0929 17:54:43.342091 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0929 17:54:43.342096 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:54:43.343298 (+ 1202us) server_negotiation.cc:407] Connection header received
0929 17:54:43.344546 (+ 1248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:54:43.344550 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:54:43.344604 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:54:43.344688 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:54:43.346056 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:54:43.346861 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:54:43.348785 (+ 1924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:54:43.349187 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:54:43.349297 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:54:43.357757 (+ 8460us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:54:43.357761 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:54:43.357764 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:54:43.357792 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:54:43.359450 (+ 1658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:54:43.360047 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:54:43.360051 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:54:43.360053 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:54:43.360144 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:54:43.360508 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:54:43.360512 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:54:43.360514 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:54:43.360750 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:54:43.360850 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0929 17:54:43.361494 (+ 644us) server_negotiation.cc:299] Negotiation successful
0929 17:54:43.361625 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 17:54:43.363088 12126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:54:43.342436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39561 (local address 127.0.98.3:41149)
0929 17:54:43.343129 (+ 693us) negotiation.cc:107] Waiting for socket to connect
0929 17:54:43.343193 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0929 17:54:43.344303 (+ 1110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:54:43.344883 (+ 580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:54:43.344893 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:54:43.345274 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:54:43.345924 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:54:43.345942 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:54:43.346999 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:54:43.347002 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:54:43.348677 (+ 1675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:54:43.348687 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:54:43.350464 (+ 1777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:54:43.350468 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:54:43.350660 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:54:43.355809 (+ 5149us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:54:43.355830 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:54:43.357559 (+ 1729us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:54:43.359578 (+ 2019us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:54:43.359585 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:54:43.359601 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:54:43.359940 (+ 339us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:54:43.360233 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:54:43.360236 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:54:43.360238 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:54:43.360408 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:54:43.360854 (+ 446us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:54:43.360860 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:54:43.361229 (+ 369us) client_negotiation.cc:769] Sending connection context
0929 17:54:43.361430 (+ 201us) client_negotiation.cc:240] Negotiation successful
0929 17:54:43.361747 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":136,"threads_started":1}
I20260929 17:54:43.364682 12124 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39561
I20260929 17:54:43.364928 12124 heartbeater.cc:464] Registering TS with master...
I20260929 17:54:43.365505 12124 heartbeater.cc:511] Master 127.0.98.62:39561 requested a full tablet report, sending...
I20260929 17:54:43.366549 11640 ts_manager.cc:194] Registered new tserver with Master: 576c9570349049bdae925f3e90b5b9e9 (127.0.98.3:40765)
I20260929 17:54:43.367268 11640 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:41149
I20260929 17:54:43.375571 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 17:54:43.390053 11858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:54:43.377983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42148 (local address 127.0.98.62:39561)
0929 17:54:43.378101 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0929 17:54:43.378106 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:54:43.381314 (+ 3208us) server_negotiation.cc:407] Connection header received
0929 17:54:43.381468 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:54:43.381472 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:54:43.381523 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:54:43.381596 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:54:43.382450 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:54:43.383278 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:54:43.384493 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:54:43.384834 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:54:43.384941 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:54:43.386577 (+ 1636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:54:43.386580 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:54:43.386583 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:54:43.386609 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:54:43.388272 (+ 1663us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:54:43.388728 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:54:43.388732 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:54:43.388734 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:54:43.388817 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:54:43.389148 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:54:43.389151 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:54:43.389153 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:54:43.389350 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:54:43.389472 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0929 17:54:43.389748 (+ 276us) server_negotiation.cc:299] Negotiation successful
0929 17:54:43.389870 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 17:54:43.393513 11640 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42148:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 17:54:43.397789 11640 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 17:54:43.417671 12136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:54:43.407509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34395 (local address 127.0.0.1:42722)
0929 17:54:43.410027 (+ 2518us) negotiation.cc:107] Waiting for socket to connect
0929 17:54:43.410083 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0929 17:54:43.410218 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:54:43.411121 (+ 903us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:54:43.411127 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:54:43.411151 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:54:43.411285 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:54:43.411291 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:54:43.412827 (+ 1536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:54:43.412831 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:54:43.414360 (+ 1529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:54:43.414372 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:54:43.417070 (+ 2698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:54:43.417075 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:54:43.417215 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:54:43.417259 (+ 44us) client_negotiation.cc:769] Sending connection context
0929 17:54:43.417369 (+ 110us) client_negotiation.cc:240] Negotiation successful
0929 17:54:43.417507 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1461,"thread_start_us":113,"threads_started":1}
I20260929 17:54:43.418370 12137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:54:43.407603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42722 (local address 127.0.98.1:34395)
0929 17:54:43.408028 (+ 425us) server_negotiation.cc:206] Beginning negotiation
0929 17:54:43.408035 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 17:54:43.410357 (+ 2322us) server_negotiation.cc:407] Connection header received
0929 17:54:43.410541 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:54:43.410547 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:54:43.410778 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:54:43.410944 (+ 166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:54:43.411488 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:54:43.412684 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:54:43.414496 (+ 1812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:54:43.415797 (+ 1301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:54:43.416821 (+ 1024us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:54:43.418055 (+ 1234us) server_negotiation.cc:1035] Waiting for connection context
0929 17:54:43.418119 (+ 64us) server_negotiation.cc:299] Negotiation successful
0929 17:54:43.418232 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":115,"threads_started":1}
I20260929 17:54:43.419868 12138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:54:43.410477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40765 (local address 127.0.0.1:40378)
0929 17:54:43.411235 (+ 758us) negotiation.cc:107] Waiting for socket to connect
0929 17:54:43.411247 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 17:54:43.411321 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:54:43.415342 (+ 4021us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:54:43.415345 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:54:43.415363 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:54:43.415447 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:54:43.415452 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:54:43.416770 (+ 1318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:54:43.416773 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:54:43.418179 (+ 1406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:54:43.418190 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:54:43.419495 (+ 1305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:54:43.419499 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:54:43.419584 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:54:43.419606 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 17:54:43.419666 (+ 60us) client_negotiation.cc:240] Negotiation successful
0929 17:54:43.419733 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":701,"mutex_wait_us":1,"thread_start_us":91,"threads_started":1}
I20260929 17:54:43.420579 12140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:54:43.409392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40378 (local address 127.0.98.3:40765)
0929 17:54:43.414685 (+ 5293us) server_negotiation.cc:206] Beginning negotiation
0929 17:54:43.414696 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 17:54:43.414721 (+ 25us) server_negotiation.cc:407] Connection header received
0929 17:54:43.414794 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:54:43.414799 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:54:43.415052 (+ 253us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:54:43.415202 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:54:43.415562 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:54:43.416632 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:54:43.418326 (+ 1694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:54:43.419213 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:54:43.419307 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:54:43.419438 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0929 17:54:43.420339 (+ 901us) server_negotiation.cc:299] Negotiation successful
0929 17:54:43.420441 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5171,"thread_start_us":115,"threads_started":1}
I20260929 17:54:43.421856 11788 tablet_service.cc:1467] Processing CreateTablet for tablet 68e5910210f54061995e90cd2b121556 (DEFAULT_TABLE table=test-table [id=c4f4d7f98ded460e9b74e2b4b6c04655]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 17:54:43.423103 11788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 68e5910210f54061995e90cd2b121556. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:54:43.423626 12057 tablet_service.cc:1467] Processing CreateTablet for tablet 68e5910210f54061995e90cd2b121556 (DEFAULT_TABLE table=test-table [id=c4f4d7f98ded460e9b74e2b4b6c04655]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 17:54:43.424603 12057 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 68e5910210f54061995e90cd2b121556. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:54:43.433102 12141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:54:43.408701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40357 (local address 127.0.0.1:53010)
0929 17:54:43.410913 (+ 2212us) negotiation.cc:107] Waiting for socket to connect
0929 17:54:43.410928 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 17:54:43.411011 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:54:43.428540 (+ 17529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:54:43.428544 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:54:43.428576 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:54:43.428714 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:54:43.428724 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:54:43.430315 (+ 1591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:54:43.430319 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:54:43.431529 (+ 1210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:54:43.431542 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:54:43.432650 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:54:43.432654 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:54:43.432757 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:54:43.432787 (+ 30us) client_negotiation.cc:769] Sending connection context
0929 17:54:43.432835 (+ 48us) client_negotiation.cc:240] Negotiation successful
0929 17:54:43.432913 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2127,"thread_start_us":88,"threads_started":1}
I20260929 17:54:43.433859 12139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:54:43.408846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53010 (local address 127.0.98.2:40357)
0929 17:54:43.418897 (+ 10051us) server_negotiation.cc:206] Beginning negotiation
0929 17:54:43.418909 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 17:54:43.418933 (+ 24us) server_negotiation.cc:407] Connection header received
0929 17:54:43.419005 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:54:43.419011 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:54:43.419216 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:54:43.428902 (+ 9686us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:54:43.429100 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:54:43.430169 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:54:43.431712 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:54:43.432507 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:54:43.433483 (+ 976us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:54:43.433604 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0929 17:54:43.433648 (+ 44us) server_negotiation.cc:299] Negotiation successful
0929 17:54:43.433726 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9935,"thread_start_us":110,"threads_started":1}
I20260929 17:54:43.434088 12142 tablet_bootstrap.cc:492] T 68e5910210f54061995e90cd2b121556 P 576c9570349049bdae925f3e90b5b9e9: Bootstrap starting.
I20260929 17:54:43.436834 12142 tablet_bootstrap.cc:654] T 68e5910210f54061995e90cd2b121556 P 576c9570349049bdae925f3e90b5b9e9: Neither blocks nor log segments found. Creating new log.
I20260929 17:54:43.436864 11924 tablet_service.cc:1467] Processing CreateTablet for tablet 68e5910210f54061995e90cd2b121556 (DEFAULT_TABLE table=test-table [id=c4f4d7f98ded460e9b74e2b4b6c04655]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 17:54:43.437767 12142 log.cc:824] T 68e5910210f54061995e90cd2b121556 P 576c9570349049bdae925f3e90b5b9e9: Log is configured to *not* fsync() on all Append() calls
I20260929 17:54:43.438035 11924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 68e5910210f54061995e90cd2b121556. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:54:43.439805 12142 tablet_bootstrap.cc:492] T 68e5910210f54061995e90cd2b121556 P 576c9570349049bdae925f3e90b5b9e9: No bootstrap required, opened a new log
I20260929 17:54:43.439993 12142 ts_tablet_manager.cc:1397] T 68e5910210f54061995e90cd2b121556 P 576c9570349049bdae925f3e90b5b9e9: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 17:54:43.441016 12143 tablet_bootstrap.cc:492] T 68e5910210f54061995e90cd2b121556 P f0ee61e1dba647f2961427e34e3c54b3: Bootstrap starting.
I20260929 17:54:43.443327 12143 tablet_bootstrap.cc:654] T 68e5910210f54061995e90cd2b121556 P f0ee61e1dba647f2961427e34e3c54b3: Neither blocks nor log segments found. Creating new log.
I20260929 17:54:43.444263 12142 raft_consensus.cc:348] T 68e5910210f54061995e90cd2b121556 P 576c9570349049bdae925f3e90b5b9e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0ee61e1dba647f2961427e34e3c54b3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34395 } } peers { permanent_uuid: "91ae4db6b5674cfe8673df38b233fe1e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40357 } } peers { permanent_uuid: "576c9570349049bdae925f3e90b5b9e9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40765 } }
I20260929 17:54:43.444641 12142 raft_consensus.cc:374] T 68e5910210f54061995e90cd2b121556 P 576c9570349049bdae925f3e90b5b9e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:54:43.444738 12142 raft_consensus.cc:729] T 68e5910210f54061995e90cd2b121556 P 576c9570349049bdae925f3e90b5b9e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 576c9570349049bdae925f3e90b5b9e9, State: Initialized, Role: FOLLOWER
I20260929 17:54:43.445312 12142 consensus_queue.cc:260] T 68e5910210f54061995e90cd2b121556 P 576c9570349049bdae925f3e90b5b9e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0ee61e1dba647f2961427e34e3c54b3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34395 } } peers { permanent_uuid: "91ae4db6b5674cfe8673df38b233fe1e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40357 } } peers { permanent_uuid: "576c9570349049bdae925f3e90b5b9e9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40765 } }
I20260929 17:54:43.446548 12142 ts_tablet_manager.cc:1428] T 68e5910210f54061995e90cd2b121556 P 576c9570349049bdae925f3e90b5b9e9: Time spent starting tablet: real 0.007s user 0.003s sys 0.004s
I20260929 17:54:43.446971 12124 heartbeater.cc:503] Master 127.0.98.62:39561 was elected leader, sending a full tablet report...
I20260929 17:54:43.448408 12146 tablet_bootstrap.cc:492] T 68e5910210f54061995e90cd2b121556 P 91ae4db6b5674cfe8673df38b233fe1e: Bootstrap starting.
I20260929 17:54:43.450815 12146 tablet_bootstrap.cc:654] T 68e5910210f54061995e90cd2b121556 P 91ae4db6b5674cfe8673df38b233fe1e: Neither blocks nor log segments found. Creating new log.
I20260929 17:54:43.452188 12146 log.cc:824] T 68e5910210f54061995e90cd2b121556 P 91ae4db6b5674cfe8673df38b233fe1e: Log is configured to *not* fsync() on all Append() calls
I20260929 17:54:43.454327 12146 tablet_bootstrap.cc:492] T 68e5910210f54061995e90cd2b121556 P 91ae4db6b5674cfe8673df38b233fe1e: No bootstrap required, opened a new log
I20260929 17:54:43.454515 12146 ts_tablet_manager.cc:1397] T 68e5910210f54061995e90cd2b121556 P 91ae4db6b5674cfe8673df38b233fe1e: Time spent bootstrapping tablet: real 0.006s user 0.006s sys 0.000s
I20260929 17:54:43.455446 12143 log.cc:824] T 68e5910210f54061995e90cd2b121556 P f0ee61e1dba647f2961427e34e3c54b3: Log is configured to *not* fsync() on all Append() calls
I20260929 17:54:43.457527 12143 tablet_bootstrap.cc:492] T 68e5910210f54061995e90cd2b121556 P f0ee61e1dba647f2961427e34e3c54b3: No bootstrap required, opened a new log
I20260929 17:54:43.457736 12143 ts_tablet_manager.cc:1397] T 68e5910210f54061995e90cd2b121556 P f0ee61e1dba647f2961427e34e3c54b3: Time spent bootstrapping tablet: real 0.017s user 0.005s sys 0.000s
I20260929 17:54:43.458776 12146 raft_consensus.cc:348] T 68e5910210f54061995e90cd2b121556 P 91ae4db6b5674cfe8673df38b233fe1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0ee61e1dba647f2961427e34e3c54b3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34395 } } peers { permanent_uuid: "91ae4db6b5674cfe8673df38b233fe1e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40357 } } peers { permanent_uuid: "576c9570349049bdae925f3e90b5b9e9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40765 } }
I20260929 17:54:43.459055 12146 raft_consensus.cc:374] T 68e5910210f54061995e90cd2b121556 P 91ae4db6b5674cfe8673df38b233fe1e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:54:43.460081 12146 raft_consensus.cc:729] T 68e5910210f54061995e90cd2b121556 P 91ae4db6b5674cfe8673df38b233fe1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91ae4db6b5674cfe8673df38b233fe1e, State: Initialized, Role: FOLLOWER
I20260929 17:54:43.460604 12146 consensus_queue.cc:260] T 68e5910210f54061995e90cd2b121556 P 91ae4db6b5674cfe8673df38b233fe1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0ee61e1dba647f2961427e34e3c54b3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34395 } } peers { permanent_uuid: "91ae4db6b5674cfe8673df38b233fe1e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40357 } } peers { permanent_uuid: "576c9570349049bdae925f3e90b5b9e9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40765 } }
I20260929 17:54:43.461709 12146 ts_tablet_manager.cc:1428] T 68e5910210f54061995e90cd2b121556 P 91ae4db6b5674cfe8673df38b233fe1e: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260929 17:54:43.462481 11990 heartbeater.cc:503] Master 127.0.98.62:39561 was elected leader, sending a full tablet report...
I20260929 17:54:43.462755 12143 raft_consensus.cc:348] T 68e5910210f54061995e90cd2b121556 P f0ee61e1dba647f2961427e34e3c54b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0ee61e1dba647f2961427e34e3c54b3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34395 } } peers { permanent_uuid: "91ae4db6b5674cfe8673df38b233fe1e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40357 } } peers { permanent_uuid: "576c9570349049bdae925f3e90b5b9e9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40765 } }
I20260929 17:54:43.463039 12143 raft_consensus.cc:374] T 68e5910210f54061995e90cd2b121556 P f0ee61e1dba647f2961427e34e3c54b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:54:43.463106 12143 raft_consensus.cc:729] T 68e5910210f54061995e90cd2b121556 P f0ee61e1dba647f2961427e34e3c54b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0ee61e1dba647f2961427e34e3c54b3, State: Initialized, Role: FOLLOWER
I20260929 17:54:43.463591 12143 consensus_queue.cc:260] T 68e5910210f54061995e90cd2b121556 P f0ee61e1dba647f2961427e34e3c54b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0ee61e1dba647f2961427e34e3c54b3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34395 } } peers { permanent_uuid: "91ae4db6b5674cfe8673df38b233fe1e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40357 } } peers { permanent_uuid: "576c9570349049bdae925f3e90b5b9e9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40765 } }
I20260929 17:54:43.464612 12143 ts_tablet_manager.cc:1428] T 68e5910210f54061995e90cd2b121556 P f0ee61e1dba647f2961427e34e3c54b3: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260929 17:54:43.464974 11854 heartbeater.cc:503] Master 127.0.98.62:39561 was elected leader, sending a full tablet report...
W20260929 17:54:43.467846 11991 tablet.cc:2369] T 68e5910210f54061995e90cd2b121556 P 91ae4db6b5674cfe8673df38b233fe1e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 17:54:43.476802 12149 raft_consensus.cc:482] T 68e5910210f54061995e90cd2b121556 P 91ae4db6b5674cfe8673df38b233fe1e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 17:54:43.477033 12149 raft_consensus.cc:504] T 68e5910210f54061995e90cd2b121556 P 91ae4db6b5674cfe8673df38b233fe1e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0ee61e1dba647f2961427e34e3c54b3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34395 } } peers { permanent_uuid: "91ae4db6b5674cfe8673df38b233fe1e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40357 } } peers { permanent_uuid: "576c9570349049bdae925f3e90b5b9e9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40765 } }
I20260929 17:54:43.480012 12149 leader_election.cc:288] T 68e5910210f54061995e90cd2b121556 P 91ae4db6b5674cfe8673df38b233fe1e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f0ee61e1dba647f2961427e34e3c54b3 (127.0.98.1:34395), 576c9570349049bdae925f3e90b5b9e9 (127.0.98.3:40765)
I20260929 17:54:43.482599 11992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:54:43.478549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34395 (local address 127.0.98.2:58813)
0929 17:54:43.478680 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0929 17:54:43.478697 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 17:54:43.478795 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:54:43.479124 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:54:43.479127 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:54:43.479149 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:54:43.479271 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:54:43.479278 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:54:43.480291 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:54:43.480295 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:54:43.481531 (+ 1236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:54:43.481540 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:54:43.482164 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:54:43.482168 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:54:43.482301 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:54:43.482324 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 17:54:43.482396 (+ 72us) client_negotiation.cc:240] Negotiation successful
0929 17:54:43.482466 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37,"mutex_wait_us":25}
I20260929 17:54:43.483139 12137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:54:43.478800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58813 (local address 127.0.98.1:34395)
0929 17:54:43.478908 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0929 17:54:43.478912 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:54:43.478926 (+ 14us) server_negotiation.cc:407] Connection header received
0929 17:54:43.478980 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:54:43.478982 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:54:43.479031 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:54:43.479110 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:54:43.479423 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:54:43.480166 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:54:43.481668 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:54:43.482059 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:54:43.482144 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:54:43.482204 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0929 17:54:43.482947 (+ 743us) server_negotiation.cc:299] Negotiation successful
0929 17:54:43.483027 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260929 17:54:43.483831 11808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68e5910210f54061995e90cd2b121556" candidate_uuid: "91ae4db6b5674cfe8673df38b233fe1e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0ee61e1dba647f2961427e34e3c54b3" is_pre_election: true
I20260929 17:54:43.484146 11808 raft_consensus.cc:2454] T 68e5910210f54061995e90cd2b121556 P f0ee61e1dba647f2961427e34e3c54b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 91ae4db6b5674cfe8673df38b233fe1e in term 0.
I20260929 17:54:43.484640 11879 leader_election.cc:302] T 68e5910210f54061995e90cd2b121556 P 91ae4db6b5674cfe8673df38b233fe1e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 91ae4db6b5674cfe8673df38b233fe1e, f0ee61e1dba647f2961427e34e3c54b3; no voters:
I20260929 17:54:43.485081 12149 raft_consensus.cc:2781] T 68e5910210f54061995e90cd2b121556 P 91ae4db6b5674cfe8673df38b233fe1e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 17:54:43.485167 12149 raft_consensus.cc:482] T 68e5910210f54061995e90cd2b121556 P 91ae4db6b5674cfe8673df38b233fe1e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 17:54:43.485208 12149 raft_consensus.cc:3037] T 68e5910210f54061995e90cd2b121556 P 91ae4db6b5674cfe8673df38b233fe1e [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:54:43.485570 12140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:54:43.479874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60363 (local address 127.0.98.3:40765)
0929 17:54:43.479980 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0929 17:54:43.479984 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:54:43.480353 (+ 369us) server_negotiation.cc:407] Connection header received
0929 17:54:43.480538 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:54:43.480541 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:54:43.480587 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:54:43.480631 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:54:43.481038 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:54:43.481746 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:54:43.484354 (+ 2608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:54:43.484930 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:54:43.485023 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:54:43.485080 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0929 17:54:43.485383 (+ 303us) server_negotiation.cc:299] Negotiation successful
0929 17:54:43.485453 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260929 17:54:43.485951 12151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:54:43.479483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40765 (local address 127.0.98.2:60363)
0929 17:54:43.480275 (+ 792us) negotiation.cc:107] Waiting for socket to connect
0929 17:54:43.480288 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 17:54:43.480415 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:54:43.480768 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:54:43.480771 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:54:43.480787 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:54:43.480893 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:54:43.480898 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:54:43.481904 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:54:43.481907 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:54:43.484232 (+ 2325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:54:43.484242 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:54:43.485179 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:54:43.485182 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:54:43.485268 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:54:43.485287 (+ 19us) client_negotiation.cc:769] Sending connection context
0929 17:54:43.485729 (+ 442us) client_negotiation.cc:240] Negotiation successful
0929 17:54:43.485797 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":723,"mutex_wait_us":4,"thread_start_us":112,"threads_started":1}
I20260929 17:54:43.486421 12064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68e5910210f54061995e90cd2b121556" candidate_uuid: "91ae4db6b5674cfe8673df38b233fe1e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "576c9570349049bdae925f3e90b5b9e9" is_pre_election: true
I20260929 17:54:43.486531 12149 raft_consensus.cc:504] T 68e5910210f54061995e90cd2b121556 P 91ae4db6b5674cfe8673df38b233fe1e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0ee61e1dba647f2961427e34e3c54b3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34395 } } peers { permanent_uuid: "91ae4db6b5674cfe8673df38b233fe1e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40357 } } peers { permanent_uuid: "576c9570349049bdae925f3e90b5b9e9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40765 } }
I20260929 17:54:43.486681 12064 raft_consensus.cc:2454] T 68e5910210f54061995e90cd2b121556 P 576c9570349049bdae925f3e90b5b9e9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 91ae4db6b5674cfe8673df38b233fe1e in term 0.
I20260929 17:54:43.487675 11808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68e5910210f54061995e90cd2b121556" candidate_uuid: "91ae4db6b5674cfe8673df38b233fe1e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0ee61e1dba647f2961427e34e3c54b3"
I20260929 17:54:43.487794 11808 raft_consensus.cc:3037] T 68e5910210f54061995e90cd2b121556 P f0ee61e1dba647f2961427e34e3c54b3 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:54:43.489068 11808 raft_consensus.cc:2454] T 68e5910210f54061995e90cd2b121556 P f0ee61e1dba647f2961427e34e3c54b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91ae4db6b5674cfe8673df38b233fe1e in term 1.
I20260929 17:54:43.489411 11879 leader_election.cc:302] T 68e5910210f54061995e90cd2b121556 P 91ae4db6b5674cfe8673df38b233fe1e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 91ae4db6b5674cfe8673df38b233fe1e, f0ee61e1dba647f2961427e34e3c54b3; no voters:
I20260929 17:54:43.490062 12064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68e5910210f54061995e90cd2b121556" candidate_uuid: "91ae4db6b5674cfe8673df38b233fe1e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "576c9570349049bdae925f3e90b5b9e9"
I20260929 17:54:43.490180 12064 raft_consensus.cc:3037] T 68e5910210f54061995e90cd2b121556 P 576c9570349049bdae925f3e90b5b9e9 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:54:43.491395 12064 raft_consensus.cc:2454] T 68e5910210f54061995e90cd2b121556 P 576c9570349049bdae925f3e90b5b9e9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91ae4db6b5674cfe8673df38b233fe1e in term 1.
I20260929 17:54:43.491765 12149 leader_election.cc:288] T 68e5910210f54061995e90cd2b121556 P 91ae4db6b5674cfe8673df38b233fe1e [CANDIDATE]: Term 1 election: Requested vote from peers f0ee61e1dba647f2961427e34e3c54b3 (127.0.98.1:34395), 576c9570349049bdae925f3e90b5b9e9 (127.0.98.3:40765)
I20260929 17:54:43.491959 12149 raft_consensus.cc:2781] T 68e5910210f54061995e90cd2b121556 P 91ae4db6b5674cfe8673df38b233fe1e [term 1 FOLLOWER]: Leader election won for term 1
I20260929 17:54:43.492086 12149 raft_consensus.cc:686] T 68e5910210f54061995e90cd2b121556 P 91ae4db6b5674cfe8673df38b233fe1e [term 1 LEADER]: Becoming Leader. State: Replica: 91ae4db6b5674cfe8673df38b233fe1e, State: Running, Role: LEADER
I20260929 17:54:43.492470 12149 consensus_queue.cc:237] T 68e5910210f54061995e90cd2b121556 P 91ae4db6b5674cfe8673df38b233fe1e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0ee61e1dba647f2961427e34e3c54b3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34395 } } peers { permanent_uuid: "91ae4db6b5674cfe8673df38b233fe1e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40357 } } peers { permanent_uuid: "576c9570349049bdae925f3e90b5b9e9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40765 } }
I20260929 17:54:43.497082 11640 catalog_manager.cc:5495] T 68e5910210f54061995e90cd2b121556 P 91ae4db6b5674cfe8673df38b233fe1e reported cstate change: term changed from 0 to 1, leader changed from <none> to 91ae4db6b5674cfe8673df38b233fe1e (127.0.98.2). New cstate: current_term: 1 leader_uuid: "91ae4db6b5674cfe8673df38b233fe1e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0ee61e1dba647f2961427e34e3c54b3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34395 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "91ae4db6b5674cfe8673df38b233fe1e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40357 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "576c9570349049bdae925f3e90b5b9e9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40765 } health_report { overall_health: UNKNOWN } } }
Sep 29 17:54:43 dist-test-slave-53nh krb5kdc[11164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704482, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260929 17:54:43.533718 12139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:54:43.528766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53024 (local address 127.0.98.2:40357)
0929 17:54:43.528899 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0929 17:54:43.528904 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:54:43.528920 (+ 16us) server_negotiation.cc:407] Connection header received
0929 17:54:43.529025 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:54:43.529029 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:54:43.529102 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:54:43.529202 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 17:54:43.529558 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:54:43.530331 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:54:43.531618 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:54:43.531942 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:54:43.532035 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:54:43.532440 (+ 405us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 17:54:43.533065 (+ 625us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 17:54:43.533184 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0929 17:54:43.533521 (+ 337us) server_negotiation.cc:299] Negotiation successful
0929 17:54:43.533579 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260929 17:54:43.540584 12064 raft_consensus.cc:1261] T 68e5910210f54061995e90cd2b121556 P 576c9570349049bdae925f3e90b5b9e9 [term 1 FOLLOWER]: Refusing update from remote peer 91ae4db6b5674cfe8673df38b233fe1e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 17:54:43.540668 11808 raft_consensus.cc:1261] T 68e5910210f54061995e90cd2b121556 P f0ee61e1dba647f2961427e34e3c54b3 [term 1 FOLLOWER]: Refusing update from remote peer 91ae4db6b5674cfe8673df38b233fe1e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 17:54:43.541320 12149 consensus_queue.cc:1035] T 68e5910210f54061995e90cd2b121556 P 91ae4db6b5674cfe8673df38b233fe1e [LEADER]: Connected to new peer: Peer: permanent_uuid: "576c9570349049bdae925f3e90b5b9e9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 17:54:43.541586 12152 consensus_queue.cc:1035] T 68e5910210f54061995e90cd2b121556 P 91ae4db6b5674cfe8673df38b233fe1e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0ee61e1dba647f2961427e34e3c54b3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 17:54:43.551525 12156 mvcc.cc:205] Tried to move back new op lower bound from 7334725564574887936 to 7334725564388872192. Current Snapshot: MvccSnapshot[applied={T|T < 7334725564574887936}]
I20260929 17:54:43.556821 12159 mvcc.cc:205] Tried to move back new op lower bound from 7334725564574887936 to 7334725564388872192. Current Snapshot: MvccSnapshot[applied={T|T < 7334725564574887936}]
I20260929 17:54:43.558036 12157 mvcc.cc:205] Tried to move back new op lower bound from 7334725564574887936 to 7334725564388872192. Current Snapshot: MvccSnapshot[applied={T|T < 7334725564574887936}]
I20260929 17:54:43.567579 11639 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42148:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 17:54:43.567770 11639 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42148:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 17:54:43.570459 11639 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 50cc925367d547a2941d686dcfe17a63: Sending DeleteTablet for 3 replicas of tablet 68e5910210f54061995e90cd2b121556
I20260929 17:54:43.571511 12057 tablet_service.cc:1514] Processing DeleteTablet for tablet 68e5910210f54061995e90cd2b121556 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 17:54:43 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40378
I20260929 17:54:43.571673 11788 tablet_service.cc:1514] Processing DeleteTablet for tablet 68e5910210f54061995e90cd2b121556 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 17:54:43 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42722
I20260929 17:54:43.572127 11924 tablet_service.cc:1514] Processing DeleteTablet for tablet 68e5910210f54061995e90cd2b121556 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 17:54:43 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53010
I20260929 17:54:43.572826 12167 tablet_replica.cc:330] T 68e5910210f54061995e90cd2b121556 P f0ee61e1dba647f2961427e34e3c54b3: stopping tablet replica
I20260929 17:54:43.573165 12167 raft_consensus.cc:2229] T 68e5910210f54061995e90cd2b121556 P f0ee61e1dba647f2961427e34e3c54b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 17:54:43.573518 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskup3Soe/build/debug/bin/kudu with pid 11721
I20260929 17:54:43.577677 12168 tablet_replica.cc:330] T 68e5910210f54061995e90cd2b121556 P 91ae4db6b5674cfe8673df38b233fe1e: stopping tablet replica
I20260929 17:54:43.577993 12168 raft_consensus.cc:2229] T 68e5910210f54061995e90cd2b121556 P 91ae4db6b5674cfe8673df38b233fe1e [term 1 LEADER]: Raft consensus shutting down.
I20260929 17:54:43.578423 12168 raft_consensus.cc:2258] T 68e5910210f54061995e90cd2b121556 P 91ae4db6b5674cfe8673df38b233fe1e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 17:54:43.581280 12166 tablet_replica.cc:330] T 68e5910210f54061995e90cd2b121556 P 576c9570349049bdae925f3e90b5b9e9: stopping tablet replica
I20260929 17:54:43.581678 12166 raft_consensus.cc:2229] T 68e5910210f54061995e90cd2b121556 P 576c9570349049bdae925f3e90b5b9e9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 17:54:43.582094 12166 raft_consensus.cc:2258] T 68e5910210f54061995e90cd2b121556 P 576c9570349049bdae925f3e90b5b9e9 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260929 17:54:43.583343 11626 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:34395 (error 108)
I20260929 17:54:43.584141 12168 ts_tablet_manager.cc:1905] T 68e5910210f54061995e90cd2b121556 P 91ae4db6b5674cfe8673df38b233fe1e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 17:54:43.584408 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskup3Soe/build/debug/bin/kudu with pid 11860
I20260929 17:54:43.585798 12166 ts_tablet_manager.cc:1905] T 68e5910210f54061995e90cd2b121556 P 576c9570349049bdae925f3e90b5b9e9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 17:54:43.588424 12166 ts_tablet_manager.cc:1918] T 68e5910210f54061995e90cd2b121556 P 576c9570349049bdae925f3e90b5b9e9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 17:54:43.588512 12166 log.cc:1196] T 68e5910210f54061995e90cd2b121556 P 576c9570349049bdae925f3e90b5b9e9: Deleting WAL directory at /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790704375975685-392-0/minicluster-data/ts-2/wal/wals/68e5910210f54061995e90cd2b121556
I20260929 17:54:43.588810 12166 ts_tablet_manager.cc:1939] T 68e5910210f54061995e90cd2b121556 P 576c9570349049bdae925f3e90b5b9e9: Deleting consensus metadata
I20260929 17:54:43.590073 11626 catalog_manager.cc:4841] TS 576c9570349049bdae925f3e90b5b9e9 (127.0.98.3:40765): tablet 68e5910210f54061995e90cd2b121556 (table test-table [id=c4f4d7f98ded460e9b74e2b4b6c04655]) successfully deleted
I20260929 17:54:43.591651 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskup3Soe/build/debug/bin/kudu with pid 11995
I20260929 17:54:43.592607 12141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:54:43.592284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34395 (local address 127.0.0.1:42734)
0929 17:54:43.592457 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0929 17:54:43.592520 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
W20260929 17:54:43.592793 11626 catalog_manager.cc:4584] TS f0ee61e1dba647f2961427e34e3c54b3 (127.0.98.1:34395): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 68e5910210f54061995e90cd2b121556: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34395: connect: Connection refused (error 111)
I20260929 17:54:43.593494 12141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:54:43.593257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40357 (local address 127.0.0.1:53032)
0929 17:54:43.593391 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0929 17:54:43.593435 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:40357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260929 17:54:43.596864 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskup3Soe/build/debug/bin/kudu with pid 11608
I20260929 17:54:43.604291 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260929 17:54:43.939117 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260929 17:54:43.939237 392 mini_ranger.cc:66] Stopping Ranger...
I20260929 17:54:44.273963 392 mini_ranger.cc:68] Stopped Ranger
2026-09-29 17:54:44.274 UTC [11194] LOG: received smart shutdown request
2026-09-29 17:54:44.277 UTC [11194] LOG: background worker "logical replication launcher" (PID 11203) exited with exit code 1
2026-09-29 17:54:44.277 UTC [11198] LOG: shutting down
2026-09-29 17:54:44.446 UTC [11194] LOG: database system is shut down
2026-09-29T17:54:44Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (44272 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704375975685-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 17:54:44 dist-test-slave-53nh krb5kdc[12179](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 17:54:44 dist-test-slave-53nh krb5kdc[12179](info): set up 2 sockets
Sep 29 17:54:44 dist-test-slave-53nh krb5kdc[12179](info): commencing operation
krb5kdc: starting...
W20260929 17:54:45.496801 392 mini_kdc.cc:121] Time spent starting KDC: real 0.933s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704375975685-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704375975685-392-0/krb5kdc/test-user.keytab.
Sep 29 17:54:45 dist-test-slave-53nh krb5kdc[12179](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704485, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T17:54:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T17:54:45Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704375975685-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704375975685-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704375975685-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704375975685-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704375975685-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704375975685-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260929 17:54:45.650791 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704375975685-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskup3Soe/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704375975685-392-0/minicluster-data/postgres -l logfile start
2026-09-29 17:54:48.050 UTC [12209] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-29 17:54:48.050 UTC [12209] LOG: listening on IPv4 address "127.0.98.20", port 51579
2026-09-29 17:54:48.053 UTC [12209] LOG: listening on Unix socket "/tmp/.s.PGSQL.51579"
2026-09-29 17:54:48.079 UTC [12212] LOG: database system was shut down at 2026-09-29 17:54:46 UTC
2026-09-29 17:54:48.083 UTC [12209] LOG: database system is ready to accept connections
I20260929 17:54:48.948647 392 mini_postgres.cc:96] Postgres bound to 51579
2026-09-29 17:54:48.954 UTC [12220] FATAL: database "slave" does not exist
127.0.98.20:51579 - accepting connections
I20260929 17:54:48.955211 392 mini_ranger.cc:161] Starting Ranger...
I20260929 17:54:48.969074 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260929 17:54:49.435045 392 mini_ranger.cc:87] Created ranger Postgres database
I20260929 17:54:49.435109 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704375975685-392-0/minicluster-data/ranger-admin
2026-09-29 17:54:49,698 [I] DB FLAVOR :POSTGRES
2026-09-29 17:54:49,699 [I] --------- Verifying Ranger DB connection ---------
2026-09-29 17:54:49,699 [I] Checking connection..
2026-09-29 17:54:49,699 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskup3Soe/build/debug/bin/postgresql.jar:/tmp/dist-test-taskup3Soe/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51579/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-29 17:54:49,952 [I] Checking connection passed.
2026-09-29 17:54:49,952 [I] --------- Verifying version history table ---------
2026-09-29 17:54:49,952 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskup3Soe/build/debug/bin/postgresql.jar:/tmp/dist-test-taskup3Soe/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51579/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 17:54:50,205 [I] Table x_db_version_h does not exist in database ranger
2026-09-29 17:54:50,205 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskup3Soe/build/debug/bin/postgresql.jar:/tmp/dist-test-taskup3Soe/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51579/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 17:54:50,448 [I] Table x_db_version_h does not exist in database ranger
2026-09-29 17:54:50,448 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-29 17:54:50,448 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskup3Soe/build/debug/bin/postgresql.jar:/tmp/dist-test-taskup3Soe/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51579/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskup3Soe/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-29 17:54:50.681 UTC [12284] WARNING: there is no transaction in progress
2026-09-29 17:54:50,686 [I] create_dbversion_catalog.sql file imported successfully
2026-09-29 17:54:50,687 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskup3Soe/build/debug/bin/postgresql.jar:/tmp/dist-test-taskup3Soe/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51579/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 17:54:50,947 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-29 17:54:50,947 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-29 17:54:50,947 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskup3Soe/build/debug/bin/postgresql.jar:/tmp/dist-test-taskup3Soe/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51579/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-29 17:54:51,188 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskup3Soe/build/debug/bin/postgresql.jar:/tmp/dist-test-taskup3Soe/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51579/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-29 17:54:51,420 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskup3Soe/build/debug/bin/postgresql.jar:/tmp/dist-test-taskup3Soe/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51579/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-53nh.c.gcp-upstream.internal','N') ;"
2026-09-29 17:54:51,635 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskup3Soe/build/debug/bin/postgresql.jar:/tmp/dist-test-taskup3Soe/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51579/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-29 17:54:51,893 [I] Table x_portal_user does not exist in database ranger
2026-09-29 17:54:51,893 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskup3Soe/build/debug/bin/postgresql.jar:/tmp/dist-test-taskup3Soe/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51579/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-29 17:54:52,141 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-29 17:54:52,141 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskup3Soe/build/debug/bin/postgresql.jar:/tmp/dist-test-taskup3Soe/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51579/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-29 17:54:52,389 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskup3Soe/build/debug/bin/postgresql.jar:/tmp/dist-test-taskup3Soe/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51579/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-29 17:54:52,624 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-29 17:54:52,624 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskup3Soe/build/debug/bin/postgresql.jar:/tmp/dist-test-taskup3Soe/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51579/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskup3Soe/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-29 17:54:53.560 UTC [12392] WARNING: there is no transaction in progress
2026-09-29 17:54:53.578 UTC [12392] WARNING: there is no transaction in progress
2026-09-29 17:54:53.594 UTC [12392] WARNING: there is no transaction in progress
2026-09-29 17:54:53.609 UTC [12392] WARNING: there is no transaction in progress
2026-09-29 17:54:53.625 UTC [12392] WARNING: there is no transaction in progress
2026-09-29 17:54:53.646 UTC [12392] WARNING: there is no transaction in progress
2026-09-29 17:54:53.711 UTC [12392] WARNING: there is no transaction in progress
2026-09-29 17:54:53.717 UTC [12392] WARNING: there is no transaction in progress
2026-09-29 17:54:53.731 UTC [12392] WARNING: there is no transaction in progress
2026-09-29 17:54:53.741 UTC [12392] WARNING: there is no transaction in progress
2026-09-29 17:54:53.751 UTC [12392] WARNING: there is no transaction in progress
2026-09-29 17:54:53.765 UTC [12392] WARNING: there is no transaction in progress
2026-09-29 17:54:53.774 UTC [12392] WARNING: there is no transaction in progress
2026-09-29 17:54:53.781 UTC [12392] WARNING: there is no transaction in progress
2026-09-29 17:54:53.787 UTC [12392] WARNING: there is no transaction in progress
2026-09-29 17:54:53.793 UTC [12392] WARNING: there is no transaction in progress
2026-09-29 17:54:54,436 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-29 17:54:54,437 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskup3Soe/build/debug/bin/postgresql.jar:/tmp/dist-test-taskup3Soe/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51579/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-29 17:54:54,680 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-29 17:54:54,681 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskup3Soe/build/debug/bin/postgresql.jar:/tmp/dist-test-taskup3Soe/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51579/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-29 17:54:54,934 [I] Table x_portal_user already exists in database 'ranger'
2026-09-29 17:54:54,934 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskup3Soe/build/debug/bin/postgresql.jar:/tmp/dist-test-taskup3Soe/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51579/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-29 17:54:55,190 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-29 17:54:55,190 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskup3Soe/build/debug/bin/postgresql.jar:/tmp/dist-test-taskup3Soe/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51579/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-29 17:54:55,442 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskup3Soe/build/debug/bin/postgresql.jar:/tmp/dist-test-taskup3Soe/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51579/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-29 17:54:55,672 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskup3Soe/build/debug/bin/postgresql.jar:/tmp/dist-test-taskup3Soe/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51579/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-53nh.c.gcp-upstream.internal';"
2026-09-29 17:54:55,913 [I] CORE_DB_SCHEMA import status has been updated
2026-09-29 17:54:55,914 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskup3Soe/build/debug/bin/postgresql.jar:/tmp/dist-test-taskup3Soe/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51579/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-29 17:54:56,147 [I] DB_PATCHES have already been applied
I20260929 17:54:56.152874 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704375975685-392-0/minicluster-data/ranger-admin:/tmp/dist-test-taskup3Soe/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskup3Soe/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskup3Soe/build/debug/bin/postgresql.jar:/tmp/dist-test-taskup3Soe/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260929 17:54:56.152941 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260929 17:54:56.156013 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:51831
Sep 29, 2026 5:54:56 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 29, 2026 5:54:56 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 29, 2026 5:54:56 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704375975685-392-0/minicluster-data/ranger-admin/ews/webapp
Sep 29, 2026 5:54:56 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704375975685-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 29, 2026 5:54:56 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704375975685-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 29, 2026 5:54:57 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704375975685-392-0/minicluster-data/ranger-admin/ews/webapp].
Sep 29, 2026 5:54:57 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260929 17:54:57.155290 392 mini_ranger.cc:160] Time spent starting Ranger: real 8.200s user 0.000s sys 0.008s
Sep 29, 2026 5:55:17 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 29, 2026 5:55:17 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 29, 2026 5:55:17 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 29, 2026 5:55:17 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 29, 2026 5:55:17 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 29, 2026 5:55:17 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 29, 2026 5:55:17 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 29, 2026 5:55:17 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 29, 2026 5:55:17 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 29, 2026 5:55:17 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 29, 2026 5:55:17 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 29, 2026 5:55:17 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 29, 2026 5:55:17 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 29, 2026 5:55:17 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 29, 2026 5:55:17 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 29, 2026 5:55:17 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 29, 2026 5:55:17 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 29, 2026 5:55:18 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260929 17:55:18.727890 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704375975685-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704375975685-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704375975685-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704375975685-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 29 17:55:18 dist-test-slave-53nh krb5kdc[12179](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704518, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260929 17:55:18.810014 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260929 17:55:18.818914 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260929 17:55:19.640450 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260929 17:55:19.640519 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704375975685-392-0/minicluster-data/ranger-kms
I20260929 17:55:19.640538 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:51579
2026-09-29 17:55:19,794 [I] DB FLAVOR :POSTGRES
2026-09-29 17:55:19,795 [I] --------- Verifying Ranger DB connection ---------
2026-09-29 17:55:19,795 [I] Checking connection
2026-09-29 17:55:19,795 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskup3Soe/build/debug/bin/postgresql.jar:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704375975685-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51579/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-29 17:55:20,023 [I] connection success
2026-09-29 17:55:20,023 [I] --------- Verifying Ranger DB tables ---------
2026-09-29 17:55:20,023 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-29 17:55:20,023 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskup3Soe/build/debug/bin/postgresql.jar:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704375975685-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51579/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-29 17:55:20,260 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-29 17:55:20,260 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-29 17:55:20,261 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-29 17:55:20,261 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskup3Soe/build/debug/bin/postgresql.jar:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704375975685-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51579/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704375975685-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-29 17:55:20,507 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260929 17:55:20.946436 392 mini_ranger_kms.cc:310] Created kms service
I20260929 17:55:21.160034 392 mini_ranger_kms.cc:326] Created kudu user
I20260929 17:55:21.203368 392 mini_ranger_kms.cc:343] Created rangerkms user
Sep 29 17:55:21 dist-test-slave-53nh krb5kdc[12179](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704518, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260929 17:55:21.579296 392 mini_ranger_kms.cc:384] Added ranger policy
I20260929 17:55:21.579448 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704375975685-392-0/minicluster-data/ranger-kms:/tmp/dist-test-taskup3Soe/build/debug/bin/postgresql.jar:/tmp/dist-test-taskup3Soe/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskup3Soe/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260929 17:55:21.579478 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260929 17:55:21.582911 392 mini_ranger_kms.cc:276] Ranger KMS PID: 12572
I20260929 17:55:21.582993 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:59339
Sep 29, 2026 5:55:22 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 29, 2026 5:55:22 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 29, 2026 5:55:22 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704375975685-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 29, 2026 5:55:22 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704375975685-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 29, 2026 5:55:23 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704375975685-392-0/minicluster-data/ranger-kms/ews/webapp].
Sep 29, 2026 5:55:23 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260929 17:55:23.663162 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 4.853s user 0.006s sys 0.132s
I20260929 17:55:23.663301 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260929 17:55:23.663388 392 mini_ranger_kms.cc:401] 127.0.98.20:59339/kms/v1/keys
Sep 29 17:55:23 dist-test-slave-53nh krb5kdc[12179](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790704523, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 29 17:55:24 dist-test-slave-53nh krb5kdc[12179](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704523, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 29 17:55:24 dist-test-slave-53nh krb5kdc[12179](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704523, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 29 17:55:26 dist-test-slave-53nh krb5kdc[12179](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704526, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704375975685-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704375975685-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 17:55:26.786900 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskup3Soe/build/debug/bin/kudu
/tmp/dist-test-taskup3Soe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704375975685-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704375975685-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704375975685-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704375975685-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46201
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704375975685-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46201
--ranger_config_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704375975685-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:59339/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704375975685-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704375975685-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704375975685-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:55:26.939393 12623 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:55:26.939692 12623 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:55:26.939731 12623 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:55:26.943897 12623 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 17:55:26.943976 12623 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:55:26.944003 12623 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:55:26.944023 12623 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 17:55:26.944073 12623 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 17:55:26.949929 12623 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704375975685-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704375975685-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:59339/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46201
--ranger_config_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704375975685-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704375975685-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46201
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704375975685-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12623
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704375975685-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:55:26.951025 12623 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:55:26.952068 12623 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:55:26.960986 12631 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:55:26.961037 12629 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:55:26.961167 12623 server_base.cc:1034] running on GCE node
W20260929 17:55:26.961262 12628 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:55:26.961958 12623 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:55:26.963079 12623 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:55:26.964270 12623 hybrid_clock.cc:648] HybridClock initialized: now 1790704526964247 us; error 43 us; skew 500 ppm
Sep 29 17:55:26 dist-test-slave-53nh krb5kdc[12179](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704526, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:55:26.967193 12623 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 17:55:26.968346 12623 webserver.cc:460] Webserver started at http://127.0.98.62:33071/ using document root <none> and password file <none>
I20260929 17:55:26.969062 12623 fs_manager.cc:359] Metadata directory not provided
I20260929 17:55:26.969143 12623 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:55:26.969383 12623 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 17:55:26 dist-test-slave-53nh krb5kdc[12179](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704526, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260929 17:55:27.017738 12623 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704375975685-392-0/minicluster-data/master-0/data/instance:
uuid: "c1c7f1bbb54d4f84afc44bb16d650329"
format_stamp: "Formatted at 2026-09-29 17:55:27 on dist-test-slave-53nh"
server_key: "c434ae95b2b173d8f59b350c16817d17"
server_key_iv: "9f15f0a62a6ca211ea41a00a2e9d2010"
server_key_version: "kuduclusterkey@0"
I20260929 17:55:27.018437 12623 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704375975685-392-0/minicluster-data/master-0/wal/instance:
uuid: "c1c7f1bbb54d4f84afc44bb16d650329"
format_stamp: "Formatted at 2026-09-29 17:55:27 on dist-test-slave-53nh"
server_key: "c434ae95b2b173d8f59b350c16817d17"
server_key_iv: "9f15f0a62a6ca211ea41a00a2e9d2010"
server_key_version: "kuduclusterkey@0"
I20260929 17:55:27.024080 12623 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260929 17:55:27.046259 12638 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:55:27.047683 12623 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 17:55:27.047823 12623 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704375975685-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704375975685-392-0/minicluster-data/master-0/wal
uuid: "c1c7f1bbb54d4f84afc44bb16d650329"
format_stamp: "Formatted at 2026-09-29 17:55:27 on dist-test-slave-53nh"
server_key: "c434ae95b2b173d8f59b350c16817d17"
server_key_iv: "9f15f0a62a6ca211ea41a00a2e9d2010"
server_key_version: "kuduclusterkey@0"
I20260929 17:55:27.047955 12623 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704375975685-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704375975685-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704375975685-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:55:27.082393 12623 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:55:27.085435 12623 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:55:27.085630 12623 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:55:27.093532 12623 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46201
I20260929 17:55:27.093537 12690 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46201 every 8 connection(s)
I20260929 17:55:27.094640 12623 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704375975685-392-0/minicluster-data/master-0/data/info.pb
I20260929 17:55:27.097579 12691 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:55:27.101913 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskup3Soe/build/debug/bin/kudu as pid 12623
I20260929 17:55:27.102056 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704375975685-392-0/minicluster-data/master-0/wal/instance
I20260929 17:55:27.102331 392 external_mini_cluster.cc:1428] Setting key ee1e84bf989b59f2dfb11f263cab573d
I20260929 17:55:27.103477 12691 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c1c7f1bbb54d4f84afc44bb16d650329: Bootstrap starting.
I20260929 17:55:27.106274 12691 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c1c7f1bbb54d4f84afc44bb16d650329: Neither blocks nor log segments found. Creating new log.
I20260929 17:55:27.107141 12691 log.cc:824] T 00000000000000000000000000000000 P c1c7f1bbb54d4f84afc44bb16d650329: Log is configured to *not* fsync() on all Append() calls
I20260929 17:55:27.109185 12691 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c1c7f1bbb54d4f84afc44bb16d650329: No bootstrap required, opened a new log
Sep 29 17:55:27 dist-test-slave-53nh krb5kdc[12179](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704526, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 17:55:27.113471 12691 raft_consensus.cc:348] T 00000000000000000000000000000000 P c1c7f1bbb54d4f84afc44bb16d650329 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1c7f1bbb54d4f84afc44bb16d650329" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46201 } }
I20260929 17:55:27.113785 12691 raft_consensus.cc:374] T 00000000000000000000000000000000 P c1c7f1bbb54d4f84afc44bb16d650329 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:55:27.113857 12691 raft_consensus.cc:729] T 00000000000000000000000000000000 P c1c7f1bbb54d4f84afc44bb16d650329 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1c7f1bbb54d4f84afc44bb16d650329, State: Initialized, Role: FOLLOWER
I20260929 17:55:27.114408 12691 consensus_queue.cc:260] T 00000000000000000000000000000000 P c1c7f1bbb54d4f84afc44bb16d650329 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1c7f1bbb54d4f84afc44bb16d650329" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46201 } }
I20260929 17:55:27.114583 12691 raft_consensus.cc:388] T 00000000000000000000000000000000 P c1c7f1bbb54d4f84afc44bb16d650329 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 17:55:27.114670 12691 raft_consensus.cc:482] T 00000000000000000000000000000000 P c1c7f1bbb54d4f84afc44bb16d650329 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 17:55:27.114773 12691 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c1c7f1bbb54d4f84afc44bb16d650329 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:55:27.115904 12691 raft_consensus.cc:504] T 00000000000000000000000000000000 P c1c7f1bbb54d4f84afc44bb16d650329 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1c7f1bbb54d4f84afc44bb16d650329" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46201 } }
I20260929 17:55:27.116263 12691 leader_election.cc:302] T 00000000000000000000000000000000 P c1c7f1bbb54d4f84afc44bb16d650329 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c1c7f1bbb54d4f84afc44bb16d650329; no voters:
I20260929 17:55:27.116657 12691 leader_election.cc:288] T 00000000000000000000000000000000 P c1c7f1bbb54d4f84afc44bb16d650329 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 17:55:27.116757 12696 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c1c7f1bbb54d4f84afc44bb16d650329 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 17:55:27.116995 12696 raft_consensus.cc:686] T 00000000000000000000000000000000 P c1c7f1bbb54d4f84afc44bb16d650329 [term 1 LEADER]: Becoming Leader. State: Replica: c1c7f1bbb54d4f84afc44bb16d650329, State: Running, Role: LEADER
I20260929 17:55:27.117375 12696 consensus_queue.cc:237] T 00000000000000000000000000000000 P c1c7f1bbb54d4f84afc44bb16d650329 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1c7f1bbb54d4f84afc44bb16d650329" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46201 } }
I20260929 17:55:27.118139 12691 sys_catalog.cc:564] T 00000000000000000000000000000000 P c1c7f1bbb54d4f84afc44bb16d650329 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 17:55:27.118724 12696 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1c7f1bbb54d4f84afc44bb16d650329 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c1c7f1bbb54d4f84afc44bb16d650329" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1c7f1bbb54d4f84afc44bb16d650329" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46201 } } }
I20260929 17:55:27.118925 12696 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1c7f1bbb54d4f84afc44bb16d650329 [sys.catalog]: This master's current role is: LEADER
I20260929 17:55:27.119199 12694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:55:27.103845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40604 (local address 127.0.98.62:46201)
0929 17:55:27.104348 (+ 503us) server_negotiation.cc:206] Beginning negotiation
0929 17:55:27.104367 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0929 17:55:27.104410 (+ 43us) server_negotiation.cc:407] Connection header received
0929 17:55:27.105300 (+ 890us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:55:27.105323 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:55:27.105746 (+ 423us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:55:27.106109 (+ 363us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:55:27.106905 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:55:27.108171 (+ 1266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:55:27.109368 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:55:27.110040 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:55:27.110198 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:55:27.113225 (+ 3027us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:55:27.113232 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:55:27.113251 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:55:27.113285 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:55:27.115267 (+ 1982us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:55:27.115782 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:55:27.115788 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:55:27.115794 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:55:27.115944 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:55:27.116296 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:55:27.116299 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:55:27.116301 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:55:27.116856 (+ 555us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:55:27.117759 (+ 903us) server_negotiation.cc:1035] Waiting for connection context
0929 17:55:27.117930 (+ 171us) server_negotiation.cc:299] Negotiation successful
0929 17:55:27.118220 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":166,"threads_started":1}
I20260929 17:55:27.119268 12696 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1c7f1bbb54d4f84afc44bb16d650329 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c1c7f1bbb54d4f84afc44bb16d650329. Latest consensus state: current_term: 1 leader_uuid: "c1c7f1bbb54d4f84afc44bb16d650329" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1c7f1bbb54d4f84afc44bb16d650329" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46201 } } }
I20260929 17:55:27.119810 12696 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1c7f1bbb54d4f84afc44bb16d650329 [sys.catalog]: This master's current role is: LEADER
I20260929 17:55:27.119546 12705 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 17:55:27.120314 12691 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704375975685-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260929 17:55:27.123198 12705 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 17:55:27.135975 12705 catalog_manager.cc:1347] Generated new cluster ID: d0e053616ffa4c49ae683ddf512fc094
I20260929 17:55:27.136108 12705 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 17:55:27.171726 12705 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 17:55:27.173192 12705 catalog_manager.cc:1504] Loading token signing keys...
I20260929 17:55:27.178524 12705 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c1c7f1bbb54d4f84afc44bb16d650329: Generated new TSK 0
I20260929 17:55:27.179486 12705 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704375975685-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704375975685-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260929 17:55:27.904139 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskup3Soe/build/debug/bin/kudu
/tmp/dist-test-taskup3Soe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704375975685-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704375975685-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704375975685-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704375975685-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:59339/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46201
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704375975685-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704375975685-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704375975685-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704375975685-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:55:28.044131 12736 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:55:28.044374 12736 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:55:28.044445 12736 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:55:28.048573 12736 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:55:28.048635 12736 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:55:28.048714 12736 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 17:55:28.054037 12736 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704375975685-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704375975685-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:59339/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704375975685-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704375975685-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46201
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12736
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704375975685-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:55:28.055025 12736 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:55:28.056013 12736 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:55:28.067957 12742 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:55:28.067951 12743 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:55:28.068571 12745 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:55:28.069837 12736 server_base.cc:1034] running on GCE node
I20260929 17:55:28.070343 12736 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:55:28.071132 12736 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:55:28.074421 12736 hybrid_clock.cc:648] HybridClock initialized: now 1790704528074339 us; error 91 us; skew 500 ppm
Sep 29 17:55:28 dist-test-slave-53nh krb5kdc[12179](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704528, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:55:28.077770 12736 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260929 17:55:28.078922 12736 webserver.cc:460] Webserver started at http://127.0.98.1:45845/ using document root <none> and password file <none>
I20260929 17:55:28.079653 12736 fs_manager.cc:359] Metadata directory not provided
I20260929 17:55:28.079746 12736 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:55:28.080061 12736 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 17:55:28 dist-test-slave-53nh krb5kdc[12179](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704528, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260929 17:55:28.101765 12736 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704375975685-392-0/minicluster-data/ts-0/data/instance:
uuid: "982fb80772c64914a0c4e12631781c30"
format_stamp: "Formatted at 2026-09-29 17:55:28 on dist-test-slave-53nh"
server_key: "fb26ce46a8eed3e272d0485b312d8dd5"
server_key_iv: "90c6a421af14a24b856da31fb4068ed9"
server_key_version: "kuduclusterkey@0"
I20260929 17:55:28.105397 12736 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704375975685-392-0/minicluster-data/ts-0/wal/instance:
uuid: "982fb80772c64914a0c4e12631781c30"
format_stamp: "Formatted at 2026-09-29 17:55:28 on dist-test-slave-53nh"
server_key: "fb26ce46a8eed3e272d0485b312d8dd5"
server_key_iv: "90c6a421af14a24b856da31fb4068ed9"
server_key_version: "kuduclusterkey@0"
I20260929 17:55:28.110349 12736 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.002s
I20260929 17:55:28.127329 12753 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:55:28.129222 12736 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 17:55:28.129364 12736 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704375975685-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704375975685-392-0/minicluster-data/ts-0/wal
uuid: "982fb80772c64914a0c4e12631781c30"
format_stamp: "Formatted at 2026-09-29 17:55:28 on dist-test-slave-53nh"
server_key: "fb26ce46a8eed3e272d0485b312d8dd5"
server_key_iv: "90c6a421af14a24b856da31fb4068ed9"
server_key_version: "kuduclusterkey@0"
I20260929 17:55:28.129500 12736 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704375975685-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704375975685-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704375975685-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:55:28.147733 12736 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:55:28.151465 12736 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:55:28.151748 12736 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:55:28.152582 12736 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 17:55:28.153736 12736 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:55:28.153851 12736 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:55:28.153975 12736 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:55:28.154045 12736 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 17:55:28.164458 12736 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40593
I20260929 17:55:28.165669 12736 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704375975685-392-0/minicluster-data/ts-0/data/info.pb
I20260929 17:55:28.172640 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskup3Soe/build/debug/bin/kudu as pid 12736
I20260929 17:55:28.172757 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704375975685-392-0/minicluster-data/ts-0/wal/instance
I20260929 17:55:28.173105 392 external_mini_cluster.cc:1428] Setting key d10ce46c82c4f9c858fa62711b07a7ff
I20260929 17:55:28.173077 12866 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40593 every 8 connection(s)
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Sep 29 17:55:28 dist-test-slave-53nh krb5kdc[12179](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704528, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260929 17:55:28.195789 12871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:55:28.175344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46201 (local address 127.0.98.1:39059)
0929 17:55:28.176156 (+ 812us) negotiation.cc:107] Waiting for socket to connect
0929 17:55:28.176218 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0929 17:55:28.177065 (+ 847us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:55:28.179454 (+ 2389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:55:28.179464 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:55:28.179808 (+ 344us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:55:28.180485 (+ 677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:55:28.180506 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:55:28.181718 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:55:28.181722 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:55:28.183115 (+ 1393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:55:28.183126 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:55:28.183837 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:55:28.183841 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:55:28.184039 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:55:28.184711 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:55:28.184731 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:55:28.188169 (+ 3438us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:55:28.192385 (+ 4216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:55:28.192392 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:55:28.192410 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:55:28.192772 (+ 362us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:55:28.193123 (+ 351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:55:28.193127 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:55:28.193128 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:55:28.193327 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:55:28.193783 (+ 456us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:55:28.193790 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:55:28.194130 (+ 340us) client_negotiation.cc:769] Sending connection context
0929 17:55:28.194340 (+ 210us) client_negotiation.cc:240] Negotiation successful
0929 17:55:28.194685 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":574,"thread_start_us":147,"threads_started":1}
I20260929 17:55:28.197278 12870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:55:28.174794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39059 (local address 127.0.98.62:46201)
0929 17:55:28.178993 (+ 4199us) server_negotiation.cc:206] Beginning negotiation
0929 17:55:28.179001 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 17:55:28.179017 (+ 16us) server_negotiation.cc:407] Connection header received
0929 17:55:28.179059 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:55:28.179063 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:55:28.179108 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:55:28.179200 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:55:28.180670 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:55:28.181569 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:55:28.183283 (+ 1714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:55:28.183664 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:55:28.185488 (+ 1824us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:55:28.188384 (+ 2896us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:55:28.188388 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:55:28.188392 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:55:28.188418 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:55:28.192214 (+ 3796us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:55:28.192895 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:55:28.192899 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:55:28.192901 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:55:28.192988 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:55:28.193443 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:55:28.193446 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:55:28.193448 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:55:28.193671 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:55:28.196836 (+ 3165us) server_negotiation.cc:1035] Waiting for connection context
0929 17:55:28.196928 (+ 92us) server_negotiation.cc:299] Negotiation successful
0929 17:55:28.197093 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4107,"thread_start_us":96,"threads_started":1}
I20260929 17:55:28.198606 12867 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46201
I20260929 17:55:28.198874 12867 heartbeater.cc:464] Registering TS with master...
I20260929 17:55:28.199501 12867 heartbeater.cc:511] Master 127.0.98.62:46201 requested a full tablet report, sending...
I20260929 17:55:28.201352 12655 ts_manager.cc:194] Registered new tserver with Master: 982fb80772c64914a0c4e12631781c30 (127.0.98.1:40593)
I20260929 17:55:28.202870 12655 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:39059
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704375975685-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704375975685-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260929 17:55:28.220355 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskup3Soe/build/debug/bin/kudu
/tmp/dist-test-taskup3Soe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704375975685-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704375975685-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704375975685-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704375975685-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:59339/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46201
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704375975685-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704375975685-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704375975685-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704375975685-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 17:55:28 dist-test-slave-53nh krb5kdc[12179](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790704528, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260929 17:55:28.374986 12875 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:55:28.375346 12875 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:55:28.375409 12875 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:55:28.381508 12875 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:55:28.381637 12875 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:55:28.381753 12875 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 17:55:28.389562 12875 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704375975685-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704375975685-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:59339/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704375975685-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704375975685-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46201
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12875
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704375975685-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:55:28.391083 12875 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:55:28.392457 12875 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:55:28.404114 12881 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:55:28.404757 12882 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:55:28.404182 12884 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:55:28.405155 12875 server_base.cc:1034] running on GCE node
I20260929 17:55:28.405678 12875 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:55:28.406564 12875 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:55:28.407791 12875 hybrid_clock.cc:648] HybridClock initialized: now 1790704528407766 us; error 43 us; skew 500 ppm
Sep 29 17:55:28 dist-test-slave-53nh krb5kdc[12179](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704528, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:55:28.411657 12875 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260929 17:55:28.412784 12875 webserver.cc:460] Webserver started at http://127.0.98.2:41271/ using document root <none> and password file <none>
I20260929 17:55:28.413398 12875 fs_manager.cc:359] Metadata directory not provided
I20260929 17:55:28.413460 12875 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:55:28.413655 12875 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 17:55:28 dist-test-slave-53nh krb5kdc[12179](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704528, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260929 17:55:28.427222 12875 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704375975685-392-0/minicluster-data/ts-1/data/instance:
uuid: "d793fe68b4284bf2b8ab8d9a4bfafaea"
format_stamp: "Formatted at 2026-09-29 17:55:28 on dist-test-slave-53nh"
server_key: "4e18e06d33be29a443f6eb359d07ebad"
server_key_iv: "2d375116be73322c2b30f9001ec5ade7"
server_key_version: "kuduclusterkey@0"
I20260929 17:55:28.428004 12875 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704375975685-392-0/minicluster-data/ts-1/wal/instance:
uuid: "d793fe68b4284bf2b8ab8d9a4bfafaea"
format_stamp: "Formatted at 2026-09-29 17:55:28 on dist-test-slave-53nh"
server_key: "4e18e06d33be29a443f6eb359d07ebad"
server_key_iv: "2d375116be73322c2b30f9001ec5ade7"
server_key_version: "kuduclusterkey@0"
I20260929 17:55:28.432240 12875 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 17:55:28.447618 12891 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:55:28.449041 12875 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 17:55:28.449172 12875 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704375975685-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704375975685-392-0/minicluster-data/ts-1/wal
uuid: "d793fe68b4284bf2b8ab8d9a4bfafaea"
format_stamp: "Formatted at 2026-09-29 17:55:28 on dist-test-slave-53nh"
server_key: "4e18e06d33be29a443f6eb359d07ebad"
server_key_iv: "2d375116be73322c2b30f9001ec5ade7"
server_key_version: "kuduclusterkey@0"
I20260929 17:55:28.449303 12875 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704375975685-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704375975685-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704375975685-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:55:28.470999 12875 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:55:28.474112 12875 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:55:28.474282 12875 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:55:28.475049 12875 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 17:55:28.475980 12875 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:55:28.476028 12875 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:55:28.476079 12875 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:55:28.476131 12875 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 17:55:28.487174 13004 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41657 every 8 connection(s)
I20260929 17:55:28.487146 12875 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41657
I20260929 17:55:28.488320 12875 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704375975685-392-0/minicluster-data/ts-1/data/info.pb
I20260929 17:55:28.491294 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskup3Soe/build/debug/bin/kudu as pid 12875
I20260929 17:55:28.491423 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704375975685-392-0/minicluster-data/ts-1/wal/instance
I20260929 17:55:28.491767 392 external_mini_cluster.cc:1428] Setting key 6432ca471994038e69dcc11fb72dc187
Sep 29 17:55:28 dist-test-slave-53nh krb5kdc[12179](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704528, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260929 17:55:28.504155 12870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:55:28.490549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56453 (local address 127.0.98.62:46201)
0929 17:55:28.490700 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0929 17:55:28.490705 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:55:28.492697 (+ 1992us) server_negotiation.cc:407] Connection header received
0929 17:55:28.494064 (+ 1367us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:55:28.494068 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:55:28.494123 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:55:28.494192 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:55:28.495870 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:55:28.496585 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:55:28.497862 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:55:28.498093 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:55:28.498187 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:55:28.500542 (+ 2355us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:55:28.500545 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:55:28.500547 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:55:28.500573 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:55:28.501996 (+ 1423us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:55:28.502501 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:55:28.502505 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:55:28.502506 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:55:28.502601 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:55:28.502931 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:55:28.502935 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:55:28.502937 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:55:28.503125 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:55:28.503218 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0929 17:55:28.503836 (+ 618us) server_negotiation.cc:299] Negotiation successful
0929 17:55:28.503972 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 17:55:28.505179 13007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:55:28.491059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46201 (local address 127.0.98.2:56453)
0929 17:55:28.492517 (+ 1458us) negotiation.cc:107] Waiting for socket to connect
0929 17:55:28.492580 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0929 17:55:28.493791 (+ 1211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:55:28.494661 (+ 870us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:55:28.494670 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:55:28.495007 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:55:28.495684 (+ 677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:55:28.495703 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:55:28.496727 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:55:28.496730 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:55:28.497730 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:55:28.497747 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:55:28.498249 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:55:28.498254 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:55:28.498431 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:55:28.499015 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:55:28.499033 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:55:28.500376 (+ 1343us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:55:28.502111 (+ 1735us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:55:28.502117 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:55:28.502133 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:55:28.502403 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:55:28.502687 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:55:28.502690 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:55:28.502692 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:55:28.502841 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:55:28.503226 (+ 385us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:55:28.503233 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:55:28.503578 (+ 345us) client_negotiation.cc:769] Sending connection context
0929 17:55:28.503783 (+ 205us) client_negotiation.cc:240] Negotiation successful
0929 17:55:28.504110 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1216,"thread_start_us":143,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260929 17:55:28.506450 13005 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46201
I20260929 17:55:28.506680 13005 heartbeater.cc:464] Registering TS with master...
I20260929 17:55:28.507191 13005 heartbeater.cc:511] Master 127.0.98.62:46201 requested a full tablet report, sending...
I20260929 17:55:28.508399 12655 ts_manager.cc:194] Registered new tserver with Master: d793fe68b4284bf2b8ab8d9a4bfafaea (127.0.98.2:41657)
I20260929 17:55:28.509142 12655 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:56453
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704375975685-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704375975685-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260929 17:55:28.520315 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskup3Soe/build/debug/bin/kudu
/tmp/dist-test-taskup3Soe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704375975685-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704375975685-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704375975685-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704375975685-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:59339/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46201
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704375975685-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704375975685-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704375975685-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704375975685-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 17:55:28 dist-test-slave-53nh krb5kdc[12179](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704528, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
W20260929 17:55:28.686723 13010 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:55:28.687150 13010 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:55:28.687240 13010 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:55:28.693110 13010 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:55:28.693253 13010 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:55:28.693403 13010 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 17:55:28.701150 13010 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704375975685-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704375975685-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:59339/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704375975685-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704375975685-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46201
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13010
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704375975685-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:55:28.702636 13010 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:55:28.704087 13010 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:55:28.716647 13015 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:55:28.716696 13016 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:55:28.716835 13018 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:55:28.717507 13010 server_base.cc:1034] running on GCE node
I20260929 17:55:28.718254 13010 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:55:28.719087 13010 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:55:28.722446 13010 hybrid_clock.cc:648] HybridClock initialized: now 1790704528722366 us; error 79 us; skew 500 ppm
Sep 29 17:55:28 dist-test-slave-53nh krb5kdc[12179](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704528, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:55:28.726069 13010 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260929 17:55:28.727134 13010 webserver.cc:460] Webserver started at http://127.0.98.3:41123/ using document root <none> and password file <none>
I20260929 17:55:28.727784 13010 fs_manager.cc:359] Metadata directory not provided
I20260929 17:55:28.727864 13010 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:55:28.728072 13010 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 17:55:28 dist-test-slave-53nh krb5kdc[12179](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704528, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260929 17:55:28.747206 13010 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704375975685-392-0/minicluster-data/ts-2/data/instance:
uuid: "9ef29865b5a34644926e8db945ba9e68"
format_stamp: "Formatted at 2026-09-29 17:55:28 on dist-test-slave-53nh"
server_key: "2561a6e0a49fb0ab22976989a0f0c045"
server_key_iv: "42ea3265432c47b4cda0000411868acb"
server_key_version: "kuduclusterkey@0"
I20260929 17:55:28.748749 13010 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704375975685-392-0/minicluster-data/ts-2/wal/instance:
uuid: "9ef29865b5a34644926e8db945ba9e68"
format_stamp: "Formatted at 2026-09-29 17:55:28 on dist-test-slave-53nh"
server_key: "2561a6e0a49fb0ab22976989a0f0c045"
server_key_iv: "42ea3265432c47b4cda0000411868acb"
server_key_version: "kuduclusterkey@0"
I20260929 17:55:28.774081 13010 fs_manager.cc:691] Time spent creating directory manager: real 0.017s user 0.004s sys 0.000s
I20260929 17:55:28.794574 13025 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:55:28.796460 13010 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.002s sys 0.000s
I20260929 17:55:28.796603 13010 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704375975685-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704375975685-392-0/minicluster-data/ts-2/wal
uuid: "9ef29865b5a34644926e8db945ba9e68"
format_stamp: "Formatted at 2026-09-29 17:55:28 on dist-test-slave-53nh"
server_key: "2561a6e0a49fb0ab22976989a0f0c045"
server_key_iv: "42ea3265432c47b4cda0000411868acb"
server_key_version: "kuduclusterkey@0"
I20260929 17:55:28.796729 13010 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704375975685-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704375975685-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704375975685-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:55:28.811533 13010 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:55:28.814525 13010 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:55:28.814744 13010 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:55:28.815336 13010 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 17:55:28.816179 13010 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:55:28.816234 13010 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:55:28.816288 13010 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:55:28.816316 13010 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 17:55:28.826537 13010 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46287
I20260929 17:55:28.827425 13010 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704375975685-392-0/minicluster-data/ts-2/data/info.pb
I20260929 17:55:28.832269 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskup3Soe/build/debug/bin/kudu as pid 13010
I20260929 17:55:28.832386 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704375975685-392-0/minicluster-data/ts-2/wal/instance
I20260929 17:55:28.832702 392 external_mini_cluster.cc:1428] Setting key 0f4b8cca8eb59a8108bd43a38adaea6f
I20260929 17:55:28.833696 13138 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46287 every 8 connection(s)
Sep 29 17:55:28 dist-test-slave-53nh krb5kdc[12179](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704528, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 17:55:28.848273 12870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:55:28.835987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33697 (local address 127.0.98.62:46201)
0929 17:55:28.836118 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0929 17:55:28.836122 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:55:28.837417 (+ 1295us) server_negotiation.cc:407] Connection header received
0929 17:55:28.838305 (+ 888us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:55:28.838308 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:55:28.838362 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:55:28.838432 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:55:28.839524 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:55:28.840099 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:55:28.841327 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:55:28.841554 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:55:28.841643 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:55:28.844221 (+ 2578us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:55:28.844225 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:55:28.844227 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:55:28.844259 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:55:28.845853 (+ 1594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:55:28.846523 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:55:28.846527 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:55:28.846528 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:55:28.846623 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:55:28.847069 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:55:28.847072 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:55:28.847074 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:55:28.847291 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:55:28.847357 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0929 17:55:28.847967 (+ 610us) server_negotiation.cc:299] Negotiation successful
0929 17:55:28.848096 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 17:55:28.849292 13141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:55:28.836459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46201 (local address 127.0.98.3:33697)
0929 17:55:28.837286 (+ 827us) negotiation.cc:107] Waiting for socket to connect
0929 17:55:28.837330 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 17:55:28.838118 (+ 788us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:55:28.838613 (+ 495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:55:28.838620 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:55:28.838895 (+ 275us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:55:28.839385 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:55:28.839407 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:55:28.840248 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:55:28.840251 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:55:28.841219 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:55:28.841229 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:55:28.841698 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:55:28.841702 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:55:28.841864 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:55:28.842598 (+ 734us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:55:28.842617 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:55:28.844041 (+ 1424us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:55:28.845988 (+ 1947us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:55:28.845995 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:55:28.846014 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:55:28.846394 (+ 380us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:55:28.846747 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:55:28.846750 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:55:28.846752 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:55:28.846959 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:55:28.847430 (+ 471us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:55:28.847435 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:55:28.847699 (+ 264us) client_negotiation.cc:769] Sending connection context
0929 17:55:28.847895 (+ 196us) client_negotiation.cc:240] Negotiation successful
0929 17:55:28.848251 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":627,"thread_start_us":123,"threads_started":1}
I20260929 17:55:28.850419 13139 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46201
I20260929 17:55:28.850659 13139 heartbeater.cc:464] Registering TS with master...
I20260929 17:55:28.851244 13139 heartbeater.cc:511] Master 127.0.98.62:46201 requested a full tablet report, sending...
I20260929 17:55:28.852319 12655 ts_manager.cc:194] Registered new tserver with Master: 9ef29865b5a34644926e8db945ba9e68 (127.0.98.3:46287)
I20260929 17:55:28.852929 12655 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:33697
I20260929 17:55:28.858760 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 17:55:28.859447 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskup3Soe/build/debug/bin/kudu with pid 12736
I20260929 17:55:28.865610 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskup3Soe/build/debug/bin/kudu with pid 12875
I20260929 17:55:28.871299 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskup3Soe/build/debug/bin/kudu with pid 13010
I20260929 17:55:28.877367 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskup3Soe/build/debug/bin/kudu with pid 12623
I20260929 17:55:28.882946 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskup3Soe/build/debug/bin/kudu
/tmp/dist-test-taskup3Soe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704375975685-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704375975685-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704375975685-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704375975685-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46201
--webserver_interface=127.0.98.62
--webserver_port=33071
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704375975685-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46201
--ranger_config_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704375975685-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:59339/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:59339/kms with env {KRB5CCNAME=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704375975685-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704375975685-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704375975685-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:55:28.991987 13142 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:55:28.992271 13142 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:55:28.992313 13142 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:55:28.996385 13142 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 17:55:28.996469 13142 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:55:28.996490 13142 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:55:28.996510 13142 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 17:55:28.996533 13142 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 17:55:29.002538 13142 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704375975685-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704375975685-392-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:59339/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46201
--ranger_config_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704375975685-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704375975685-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46201
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704375975685-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=33071
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13142
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704375975685-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:55:29.003669 13142 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:55:29.004703 13142 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:55:29.013247 13147 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:55:29.013262 13148 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:55:29.013438 13142 server_base.cc:1034] running on GCE node
W20260929 17:55:29.013262 13150 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:55:29.014058 13142 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:55:29.015180 13142 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:55:29.016342 13142 hybrid_clock.cc:648] HybridClock initialized: now 1790704529016324 us; error 33 us; skew 500 ppm
Sep 29 17:55:29 dist-test-slave-53nh krb5kdc[12179](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704529, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:55:29.019001 13142 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 17:55:29.020107 13142 webserver.cc:460] Webserver started at http://127.0.98.62:33071/ using document root <none> and password file <none>
I20260929 17:55:29.020912 13142 fs_manager.cc:359] Metadata directory not provided
I20260929 17:55:29.021034 13142 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 29 17:55:29 dist-test-slave-53nh krb5kdc[12179](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704529, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260929 17:55:29.090210 13142 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260929 17:55:29.092402 13158 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:55:29.093696 13142 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 17:55:29.093813 13142 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704375975685-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704375975685-392-0/minicluster-data/master-0/wal
uuid: "c1c7f1bbb54d4f84afc44bb16d650329"
format_stamp: "Formatted at 2026-09-29 17:55:27 on dist-test-slave-53nh"
server_key: "c434ae95b2b173d8f59b350c16817d17"
server_key_iv: "9f15f0a62a6ca211ea41a00a2e9d2010"
server_key_version: "kuduclusterkey@0"
I20260929 17:55:29.094369 13142 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704375975685-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704375975685-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704375975685-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:55:29.122555 13142 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:55:29.125514 13142 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:55:29.125696 13142 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:55:29.132548 13142 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46201
I20260929 17:55:29.132572 13210 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46201 every 8 connection(s)
I20260929 17:55:29.133669 13142 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704375975685-392-0/minicluster-data/master-0/data/info.pb
I20260929 17:55:29.137869 13211 sys_catalog.cc:263] Verifying existing consensus state
I20260929 17:55:29.139613 13211 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c1c7f1bbb54d4f84afc44bb16d650329: Bootstrap starting.
I20260929 17:55:29.139945 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskup3Soe/build/debug/bin/kudu as pid 13142
I20260929 17:55:29.140337 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskup3Soe/build/debug/bin/kudu
/tmp/dist-test-taskup3Soe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704375975685-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704375975685-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704375975685-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704375975685-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:59339/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:40593
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:46201
--webserver_port=45845
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704375975685-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:59339/kms with env {KRB5CCNAME=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704375975685-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704375975685-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704375975685-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 17:55:29.156097 13211 log.cc:824] T 00000000000000000000000000000000 P c1c7f1bbb54d4f84afc44bb16d650329: Log is configured to *not* fsync() on all Append() calls
I20260929 17:55:29.162500 13211 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c1c7f1bbb54d4f84afc44bb16d650329: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 17:55:29.162854 13211 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c1c7f1bbb54d4f84afc44bb16d650329: Bootstrap complete.
I20260929 17:55:29.166816 13211 raft_consensus.cc:348] T 00000000000000000000000000000000 P c1c7f1bbb54d4f84afc44bb16d650329 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1c7f1bbb54d4f84afc44bb16d650329" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46201 } }
I20260929 17:55:29.167192 13211 raft_consensus.cc:729] T 00000000000000000000000000000000 P c1c7f1bbb54d4f84afc44bb16d650329 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1c7f1bbb54d4f84afc44bb16d650329, State: Initialized, Role: FOLLOWER
I20260929 17:55:29.167650 13211 consensus_queue.cc:260] T 00000000000000000000000000000000 P c1c7f1bbb54d4f84afc44bb16d650329 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1c7f1bbb54d4f84afc44bb16d650329" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46201 } }
I20260929 17:55:29.167764 13211 raft_consensus.cc:388] T 00000000000000000000000000000000 P c1c7f1bbb54d4f84afc44bb16d650329 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 17:55:29.167826 13211 raft_consensus.cc:482] T 00000000000000000000000000000000 P c1c7f1bbb54d4f84afc44bb16d650329 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 17:55:29.167915 13211 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c1c7f1bbb54d4f84afc44bb16d650329 [term 1 FOLLOWER]: Advancing to term 2
I20260929 17:55:29.169752 13211 raft_consensus.cc:504] T 00000000000000000000000000000000 P c1c7f1bbb54d4f84afc44bb16d650329 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1c7f1bbb54d4f84afc44bb16d650329" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46201 } }
I20260929 17:55:29.170064 13211 leader_election.cc:302] T 00000000000000000000000000000000 P c1c7f1bbb54d4f84afc44bb16d650329 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c1c7f1bbb54d4f84afc44bb16d650329; no voters:
I20260929 17:55:29.170363 13211 leader_election.cc:288] T 00000000000000000000000000000000 P c1c7f1bbb54d4f84afc44bb16d650329 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260929 17:55:29.170487 13215 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c1c7f1bbb54d4f84afc44bb16d650329 [term 2 FOLLOWER]: Leader election won for term 2
I20260929 17:55:29.170693 13215 raft_consensus.cc:686] T 00000000000000000000000000000000 P c1c7f1bbb54d4f84afc44bb16d650329 [term 2 LEADER]: Becoming Leader. State: Replica: c1c7f1bbb54d4f84afc44bb16d650329, State: Running, Role: LEADER
I20260929 17:55:29.170993 13215 consensus_queue.cc:237] T 00000000000000000000000000000000 P c1c7f1bbb54d4f84afc44bb16d650329 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1c7f1bbb54d4f84afc44bb16d650329" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46201 } }
I20260929 17:55:29.171583 13211 sys_catalog.cc:564] T 00000000000000000000000000000000 P c1c7f1bbb54d4f84afc44bb16d650329 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 17:55:29.171927 13215 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1c7f1bbb54d4f84afc44bb16d650329 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c1c7f1bbb54d4f84afc44bb16d650329. Latest consensus state: current_term: 2 leader_uuid: "c1c7f1bbb54d4f84afc44bb16d650329" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1c7f1bbb54d4f84afc44bb16d650329" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46201 } } }
I20260929 17:55:29.172030 13215 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1c7f1bbb54d4f84afc44bb16d650329 [sys.catalog]: This master's current role is: LEADER
I20260929 17:55:29.172297 13216 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1c7f1bbb54d4f84afc44bb16d650329 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "c1c7f1bbb54d4f84afc44bb16d650329" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1c7f1bbb54d4f84afc44bb16d650329" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46201 } } }
I20260929 17:55:29.172392 13216 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1c7f1bbb54d4f84afc44bb16d650329 [sys.catalog]: This master's current role is: LEADER
I20260929 17:55:29.172420 13219 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 17:55:29.175556 13219 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 17:55:29.176096 13211 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704375975685-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260929 17:55:29.176162 13219 catalog_manager.cc:1259] Loaded cluster ID: d0e053616ffa4c49ae683ddf512fc094
I20260929 17:55:29.176210 13219 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 17:55:29.178539 13219 catalog_manager.cc:1504] Loading token signing keys...
I20260929 17:55:29.179181 13219 catalog_manager.cc:5879] T 00000000000000000000000000000000 P c1c7f1bbb54d4f84afc44bb16d650329: Loaded TSK: 0
I20260929 17:55:29.180371 13219 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 17:55:29.278977 13213 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:55:29.279356 13213 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:55:29.279419 13213 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:55:29.285557 13213 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:55:29.285641 13213 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:55:29.285729 13213 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 17:55:29.293519 13213 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704375975685-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704375975685-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:59339/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704375975685-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:40593
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704375975685-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=45845
--tserver_master_addrs=127.0.98.62:46201
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13213
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704375975685-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:55:29.294896 13213 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:55:29.296172 13213 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:55:29.308851 13251 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:55:29.308871 13248 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:55:29.309688 13213 server_base.cc:1034] running on GCE node
W20260929 17:55:29.309996 13249 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:55:29.312740 13213 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:55:29.313565 13213 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:55:29.314769 13213 hybrid_clock.cc:648] HybridClock initialized: now 1790704529314733 us; error 53 us; skew 500 ppm
Sep 29 17:55:29 dist-test-slave-53nh krb5kdc[12179](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704529, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:55:29.318423 13213 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260929 17:55:29.319563 13213 webserver.cc:460] Webserver started at http://127.0.98.1:45845/ using document root <none> and password file <none>
I20260929 17:55:29.320286 13213 fs_manager.cc:359] Metadata directory not provided
I20260929 17:55:29.320395 13213 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 29 17:55:29 dist-test-slave-53nh krb5kdc[12179](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704529, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260929 17:55:29.350593 13213 fs_manager.cc:709] Time spent opening directory manager: real 0.008s user 0.003s sys 0.000s
I20260929 17:55:29.353978 13259 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:55:29.355638 13213 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 17:55:29.355791 13213 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704375975685-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704375975685-392-0/minicluster-data/ts-0/wal
uuid: "982fb80772c64914a0c4e12631781c30"
format_stamp: "Formatted at 2026-09-29 17:55:28 on dist-test-slave-53nh"
server_key: "fb26ce46a8eed3e272d0485b312d8dd5"
server_key_iv: "90c6a421af14a24b856da31fb4068ed9"
server_key_version: "kuduclusterkey@0"
I20260929 17:55:29.356547 13213 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704375975685-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704375975685-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704375975685-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:55:29.377024 13213 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:55:29.380632 13213 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:55:29.380843 13213 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:55:29.381582 13213 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 17:55:29.382615 13213 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:55:29.382678 13213 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:55:29.382738 13213 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:55:29.382769 13213 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 17:55:29.393399 13213 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40593
I20260929 17:55:29.393826 13372 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40593 every 8 connection(s)
I20260929 17:55:29.394703 13213 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704375975685-392-0/minicluster-data/ts-0/data/info.pb
I20260929 17:55:29.399991 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskup3Soe/build/debug/bin/kudu as pid 13213
I20260929 17:55:29.400481 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskup3Soe/build/debug/bin/kudu
/tmp/dist-test-taskup3Soe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704375975685-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704375975685-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704375975685-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704375975685-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:59339/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:41657
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:46201
--webserver_port=41271
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704375975685-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:59339/kms with env {KRB5CCNAME=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704375975685-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704375975685-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704375975685-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 17:55:29 dist-test-slave-53nh krb5kdc[12179](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704529, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 17:55:29.419325 13376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:55:29.397342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46201 (local address 127.0.98.1:52893)
0929 17:55:29.398132 (+ 790us) negotiation.cc:107] Waiting for socket to connect
0929 17:55:29.398189 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0929 17:55:29.399417 (+ 1228us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:55:29.401891 (+ 2474us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:55:29.401901 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:55:29.402249 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:55:29.402919 (+ 670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:55:29.402941 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:55:29.404817 (+ 1876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:55:29.404820 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:55:29.406280 (+ 1460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:55:29.406293 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:55:29.407654 (+ 1361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:55:29.407659 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:55:29.407873 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:55:29.408618 (+ 745us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:55:29.408640 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:55:29.412800 (+ 4160us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:55:29.415547 (+ 2747us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:55:29.415554 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:55:29.415572 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:55:29.415960 (+ 388us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:55:29.416369 (+ 409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:55:29.416372 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:55:29.416374 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:55:29.416537 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:55:29.417366 (+ 829us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:55:29.417373 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:55:29.417693 (+ 320us) client_negotiation.cc:769] Sending connection context
0929 17:55:29.417897 (+ 204us) client_negotiation.cc:240] Negotiation successful
0929 17:55:29.418258 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":568,"thread_start_us":137,"threads_started":1}
I20260929 17:55:29.419385 13375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:55:29.397199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52893 (local address 127.0.98.62:46201)
0929 17:55:29.397801 (+ 602us) server_negotiation.cc:206] Beginning negotiation
0929 17:55:29.397824 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0929 17:55:29.398774 (+ 950us) server_negotiation.cc:407] Connection header received
0929 17:55:29.400027 (+ 1253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:55:29.400041 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:55:29.400756 (+ 715us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:55:29.401166 (+ 410us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:55:29.403153 (+ 1987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:55:29.404646 (+ 1493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:55:29.406434 (+ 1788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:55:29.407216 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:55:29.407367 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:55:29.413152 (+ 5785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:55:29.413160 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:55:29.413182 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:55:29.413223 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:55:29.415365 (+ 2142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:55:29.416103 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:55:29.416110 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:55:29.416117 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:55:29.416266 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:55:29.416637 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:55:29.416640 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:55:29.416642 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:55:29.417179 (+ 537us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:55:29.417298 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0929 17:55:29.418111 (+ 813us) server_negotiation.cc:299] Negotiation successful
0929 17:55:29.418441 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":438,"thread_start_us":108,"threads_started":1}
I20260929 17:55:29.422786 13373 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46201
I20260929 17:55:29.423058 13373 heartbeater.cc:464] Registering TS with master...
I20260929 17:55:29.423652 13373 heartbeater.cc:511] Master 127.0.98.62:46201 requested a full tablet report, sending...
W20260929 17:55:29.425943 13373 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:46201 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260929 17:55:29.426074 13373 heartbeater.cc:464] Registering TS with master...
I20260929 17:55:29.426257 13373 heartbeater.cc:511] Master 127.0.98.62:46201 requested a full tablet report, sending...
I20260929 17:55:29.427088 13373 heartbeater.cc:464] Registering TS with master...
I20260929 17:55:29.427263 13373 heartbeater.cc:511] Master 127.0.98.62:46201 requested a full tablet report, sending...
W20260929 17:55:29.428045 13373 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 17:55:29.534025 13377 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:55:29.534303 13377 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:55:29.534371 13377 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:55:29.539009 13377 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:55:29.539083 13377 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:55:29.539166 13377 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 17:55:29.544602 13377 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704375975685-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704375975685-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:59339/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704375975685-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:41657
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704375975685-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=41271
--tserver_master_addrs=127.0.98.62:46201
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13377
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704375975685-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:55:29.545722 13377 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:55:29.546763 13377 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:55:29.559396 13382 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:55:29.559494 13385 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:55:29.559767 13383 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:55:29.560401 13377 server_base.cc:1034] running on GCE node
I20260929 17:55:29.560834 13377 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:55:29.561655 13377 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:55:29.564973 13377 hybrid_clock.cc:648] HybridClock initialized: now 1790704529564904 us; error 76 us; skew 500 ppm
Sep 29 17:55:29 dist-test-slave-53nh krb5kdc[12179](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704529, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:55:29.569056 13377 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260929 17:55:29.570283 13377 webserver.cc:460] Webserver started at http://127.0.98.2:41271/ using document root <none> and password file <none>
I20260929 17:55:29.571117 13377 fs_manager.cc:359] Metadata directory not provided
I20260929 17:55:29.571270 13377 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 29 17:55:29 dist-test-slave-53nh krb5kdc[12179](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704529, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260929 17:55:29.595854 13377 fs_manager.cc:709] Time spent opening directory manager: real 0.007s user 0.003s sys 0.000s
I20260929 17:55:29.599229 13393 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:55:29.600660 13377 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 17:55:29.600788 13377 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704375975685-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704375975685-392-0/minicluster-data/ts-1/wal
uuid: "d793fe68b4284bf2b8ab8d9a4bfafaea"
format_stamp: "Formatted at 2026-09-29 17:55:28 on dist-test-slave-53nh"
server_key: "4e18e06d33be29a443f6eb359d07ebad"
server_key_iv: "2d375116be73322c2b30f9001ec5ade7"
server_key_version: "kuduclusterkey@0"
I20260929 17:55:29.601428 13377 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704375975685-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704375975685-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704375975685-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:55:29.625526 13377 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:55:29.629565 13377 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:55:29.629778 13377 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:55:29.630445 13377 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 17:55:29.631443 13377 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:55:29.631488 13377 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:55:29.631534 13377 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:55:29.631554 13377 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 17:55:29.643890 13377 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41657
I20260929 17:55:29.643959 13507 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41657 every 8 connection(s)
I20260929 17:55:29.645320 13377 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704375975685-392-0/minicluster-data/ts-1/data/info.pb
I20260929 17:55:29.649183 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskup3Soe/build/debug/bin/kudu as pid 13377
I20260929 17:55:29.649636 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskup3Soe/build/debug/bin/kudu
/tmp/dist-test-taskup3Soe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704375975685-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704375975685-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704375975685-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704375975685-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:59339/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:46287
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:46201
--webserver_port=41123
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704375975685-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:59339/kms with env {KRB5CCNAME=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704375975685-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704375975685-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704375975685-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 17:55:29 dist-test-slave-53nh krb5kdc[12179](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704529, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 17:55:29.660740 13375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:55:29.647350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52305 (local address 127.0.98.62:46201)
0929 17:55:29.647503 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0929 17:55:29.647508 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:55:29.648355 (+ 847us) server_negotiation.cc:407] Connection header received
0929 17:55:29.649435 (+ 1080us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:55:29.649439 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:55:29.649504 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:55:29.649608 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:55:29.651008 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:55:29.651581 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:55:29.653222 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:55:29.653567 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:55:29.653680 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:55:29.656334 (+ 2654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:55:29.656338 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:55:29.656341 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:55:29.656376 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:55:29.658159 (+ 1783us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:55:29.659087 (+ 928us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:55:29.659090 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:55:29.659092 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:55:29.659178 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:55:29.659545 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:55:29.659548 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:55:29.659550 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:55:29.659793 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:55:29.659889 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0929 17:55:29.660436 (+ 547us) server_negotiation.cc:299] Negotiation successful
0929 17:55:29.660571 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 17:55:29.661746 13510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:55:29.647713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46201 (local address 127.0.98.2:52305)
0929 17:55:29.648177 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0929 17:55:29.648227 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0929 17:55:29.649196 (+ 969us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:55:29.649836 (+ 640us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:55:29.649847 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:55:29.650175 (+ 328us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:55:29.650818 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:55:29.650845 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:55:29.651703 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:55:29.651706 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:55:29.653084 (+ 1378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:55:29.653095 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:55:29.653750 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:55:29.653755 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:55:29.653963 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:55:29.654630 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:55:29.654651 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:55:29.656140 (+ 1489us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:55:29.658677 (+ 2537us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:55:29.658683 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:55:29.658697 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:55:29.658978 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:55:29.659267 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:55:29.659269 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:55:29.659271 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:55:29.659453 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:55:29.659900 (+ 447us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:55:29.659907 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:55:29.660199 (+ 292us) client_negotiation.cc:769] Sending connection context
0929 17:55:29.660377 (+ 178us) client_negotiation.cc:240] Negotiation successful
0929 17:55:29.660728 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":110,"threads_started":1}
I20260929 17:55:29.662945 13508 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46201
I20260929 17:55:29.663185 13508 heartbeater.cc:464] Registering TS with master...
I20260929 17:55:29.663698 13508 heartbeater.cc:511] Master 127.0.98.62:46201 requested a full tablet report, sending...
W20260929 17:55:29.665177 13508 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:46201 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260929 17:55:29.665292 13508 heartbeater.cc:464] Registering TS with master...
I20260929 17:55:29.665449 13508 heartbeater.cc:511] Master 127.0.98.62:46201 requested a full tablet report, sending...
I20260929 17:55:29.666174 13508 heartbeater.cc:464] Registering TS with master...
I20260929 17:55:29.666330 13508 heartbeater.cc:511] Master 127.0.98.62:46201 requested a full tablet report, sending...
W20260929 17:55:29.666975 13508 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 17:55:29.806043 13511 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:55:29.806429 13511 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:55:29.806497 13511 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:55:29.812635 13511 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:55:29.812713 13511 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:55:29.812799 13511 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 17:55:29.818717 13511 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704375975685-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704375975685-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:59339/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704375975685-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:46287
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704375975685-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=41123
--tserver_master_addrs=127.0.98.62:46201
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13511
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704375975685-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:55:29.819767 13511 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:55:29.820793 13511 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:55:29.832340 13517 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:55:29.832392 13519 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:55:29.833101 13516 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:55:29.833259 13511 server_base.cc:1034] running on GCE node
I20260929 17:55:29.833735 13511 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:55:29.834564 13511 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:55:29.835755 13511 hybrid_clock.cc:648] HybridClock initialized: now 1790704529835749 us; error 81 us; skew 500 ppm
Sep 29 17:55:29 dist-test-slave-53nh krb5kdc[12179](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704529, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:55:29.838635 13511 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260929 17:55:29.839830 13511 webserver.cc:460] Webserver started at http://127.0.98.3:41123/ using document root <none> and password file <none>
I20260929 17:55:29.840597 13511 fs_manager.cc:359] Metadata directory not provided
I20260929 17:55:29.840713 13511 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 29 17:55:29 dist-test-slave-53nh krb5kdc[12179](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704529, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260929 17:55:29.865873 13511 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.003s sys 0.000s
I20260929 17:55:29.874737 13527 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:55:29.876449 13511 fs_manager.cc:725] Time spent opening block manager: real 0.009s user 0.002s sys 0.000s
I20260929 17:55:29.876619 13511 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704375975685-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704375975685-392-0/minicluster-data/ts-2/wal
uuid: "9ef29865b5a34644926e8db945ba9e68"
format_stamp: "Formatted at 2026-09-29 17:55:28 on dist-test-slave-53nh"
server_key: "2561a6e0a49fb0ab22976989a0f0c045"
server_key_iv: "42ea3265432c47b4cda0000411868acb"
server_key_version: "kuduclusterkey@0"
I20260929 17:55:29.877440 13511 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704375975685-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704375975685-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704375975685-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:55:29.899890 13511 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:55:29.903582 13511 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:55:29.903813 13511 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:55:29.904652 13511 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 17:55:29.905782 13511 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:55:29.905838 13511 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:55:29.905896 13511 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:55:29.905927 13511 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 17:55:29.916628 13511 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46287
I20260929 17:55:29.916674 13647 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46287 every 8 connection(s)
I20260929 17:55:29.918231 13511 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704375975685-392-0/minicluster-data/ts-2/data/info.pb
I20260929 17:55:29.922559 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskup3Soe/build/debug/bin/kudu as pid 13511
Sep 29 17:55:29 dist-test-slave-53nh krb5kdc[12179](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704529, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 17:55:29.934103 13375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:55:29.920252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55767 (local address 127.0.98.62:46201)
0929 17:55:29.920433 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0929 17:55:29.920438 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:55:29.921313 (+ 875us) server_negotiation.cc:407] Connection header received
0929 17:55:29.922300 (+ 987us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:55:29.922304 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:55:29.922360 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:55:29.922472 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:55:29.924228 (+ 1756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:55:29.925113 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:55:29.926720 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:55:29.926945 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:55:29.927036 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:55:29.929805 (+ 2769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:55:29.929810 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:55:29.929813 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:55:29.929878 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:55:29.931411 (+ 1533us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:55:29.932013 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:55:29.932017 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:55:29.932019 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:55:29.932165 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:55:29.932850 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:55:29.932854 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:55:29.932856 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:55:29.933116 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:55:29.933249 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0929 17:55:29.933824 (+ 575us) server_negotiation.cc:299] Negotiation successful
0929 17:55:29.933938 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71,"spinlock_wait_cycles":65408}
I20260929 17:55:29.935155 13650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:55:29.920639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46201 (local address 127.0.98.3:55767)
0929 17:55:29.921152 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0929 17:55:29.921205 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0929 17:55:29.922090 (+ 885us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:55:29.922629 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:55:29.922635 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:55:29.923051 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:55:29.924047 (+ 996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:55:29.924069 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:55:29.925269 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:55:29.925274 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:55:29.926607 (+ 1333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:55:29.926616 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:55:29.927097 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:55:29.927102 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:55:29.927296 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:55:29.927991 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:55:29.928015 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:55:29.929618 (+ 1603us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:55:29.931527 (+ 1909us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:55:29.931533 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:55:29.931549 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:55:29.931928 (+ 379us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:55:29.932274 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:55:29.932277 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:55:29.932279 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:55:29.932475 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:55:29.933255 (+ 780us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:55:29.933262 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:55:29.933593 (+ 331us) client_negotiation.cc:769] Sending connection context
0929 17:55:29.933813 (+ 220us) client_negotiation.cc:240] Negotiation successful
0929 17:55:29.934129 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":126,"threads_started":1}
I20260929 17:55:29.935984 13651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:55:29.923202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33714 (local address 127.0.98.62:46201)
0929 17:55:29.926164 (+ 2962us) server_negotiation.cc:206] Beginning negotiation
0929 17:55:29.926173 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 17:55:29.926188 (+ 15us) server_negotiation.cc:407] Connection header received
0929 17:55:29.926228 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:55:29.926230 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:55:29.926290 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:55:29.926387 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:55:29.927296 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:55:29.927970 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:55:29.929345 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:55:29.929674 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:55:29.929774 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:55:29.931641 (+ 1867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:55:29.931646 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:55:29.931649 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:55:29.931673 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:55:29.933417 (+ 1744us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:55:29.934334 (+ 917us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:55:29.934337 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:55:29.934339 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:55:29.934419 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:55:29.934710 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:55:29.934712 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:55:29.934714 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:55:29.934904 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:55:29.935111 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0929 17:55:29.935709 (+ 598us) server_negotiation.cc:299] Negotiation successful
0929 17:55:29.935824 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2884,"thread_start_us":116,"threads_started":1}
I20260929 17:55:29.936539 13648 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46201
I20260929 17:55:29.936796 13648 heartbeater.cc:464] Registering TS with master...
I20260929 17:55:29.937361 13648 heartbeater.cc:511] Master 127.0.98.62:46201 requested a full tablet report, sending...
I20260929 17:55:29.938784 13175 ts_manager.cc:194] Registered new tserver with Master: 9ef29865b5a34644926e8db945ba9e68 (127.0.98.3:46287)
I20260929 17:55:29.940316 13175 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:55767
Sep 29 17:55:30 dist-test-slave-53nh krb5kdc[12179](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790704530, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:55:30.428262 13373 heartbeater.cc:464] Registering TS with master...
I20260929 17:55:30.428570 13373 heartbeater.cc:511] Master 127.0.98.62:46201 requested a full tablet report, sending...
I20260929 17:55:30.429350 13174 ts_manager.cc:194] Registered new tserver with Master: 982fb80772c64914a0c4e12631781c30 (127.0.98.1:40593)
I20260929 17:55:30.430181 13174 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:52893
I20260929 17:55:30.667172 13508 heartbeater.cc:464] Registering TS with master...
I20260929 17:55:30.667456 13508 heartbeater.cc:511] Master 127.0.98.62:46201 requested a full tablet report, sending...
I20260929 17:55:30.668206 13174 ts_manager.cc:194] Registered new tserver with Master: d793fe68b4284bf2b8ab8d9a4bfafaea (127.0.98.2:41657)
I20260929 17:55:30.669023 13174 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:52305
I20260929 17:55:30.678110 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 17:55:30 dist-test-slave-53nh krb5kdc[12179](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704530, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260929 17:55:30.689148 13663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:55:30.679746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33716 (local address 127.0.98.62:46201)
0929 17:55:30.680328 (+ 582us) server_negotiation.cc:206] Beginning negotiation
0929 17:55:30.680332 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:55:30.680344 (+ 12us) server_negotiation.cc:407] Connection header received
0929 17:55:30.680379 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:55:30.680382 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:55:30.680425 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:55:30.680506 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:55:30.681332 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:55:30.682148 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:55:30.683298 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:55:30.683706 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:55:30.683796 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:55:30.685457 (+ 1661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:55:30.685461 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:55:30.685464 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:55:30.685489 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:55:30.687116 (+ 1627us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:55:30.687572 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:55:30.687575 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:55:30.687577 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:55:30.687656 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:55:30.687914 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:55:30.687917 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:55:30.687918 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:55:30.688113 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:55:30.688227 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0929 17:55:30.688831 (+ 604us) server_negotiation.cc:299] Negotiation successful
0929 17:55:30.688947 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":498,"thread_start_us":87,"threads_started":1}
I20260929 17:55:30.692970 13174 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33716:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 17:55:30.697408 13174 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 17:55:30.728132 13668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:55:30.716793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40593 (local address 127.0.0.1:51194)
0929 17:55:30.721201 (+ 4408us) negotiation.cc:107] Waiting for socket to connect
0929 17:55:30.721214 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 17:55:30.721300 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:55:30.722137 (+ 837us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:55:30.722755 (+ 618us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:55:30.722775 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:55:30.722866 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:55:30.722872 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:55:30.724233 (+ 1361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:55:30.724236 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:55:30.725392 (+ 1156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:55:30.725403 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:55:30.727557 (+ 2154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:55:30.727563 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:55:30.727678 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:55:30.727721 (+ 43us) client_negotiation.cc:769] Sending connection context
0929 17:55:30.727839 (+ 118us) client_negotiation.cc:240] Negotiation successful
0929 17:55:30.727969 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4346,"thread_start_us":83,"threads_started":1}
I20260929 17:55:30.728870 13673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:55:30.716982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51194 (local address 127.0.98.1:40593)
0929 17:55:30.721530 (+ 4548us) server_negotiation.cc:206] Beginning negotiation
0929 17:55:30.721537 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 17:55:30.721559 (+ 22us) server_negotiation.cc:407] Connection header received
0929 17:55:30.721628 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:55:30.721634 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:55:30.721827 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:55:30.721963 (+ 136us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:55:30.722992 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:55:30.724102 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:55:30.725527 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:55:30.726344 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:55:30.726449 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:55:30.726612 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0929 17:55:30.728626 (+ 2014us) server_negotiation.cc:299] Negotiation successful
0929 17:55:30.728740 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4441,"thread_start_us":67,"threads_started":1}
I20260929 17:55:30.730199 13672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:55:30.715059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46287 (local address 127.0.0.1:60648)
0929 17:55:30.719338 (+ 4279us) negotiation.cc:107] Waiting for socket to connect
0929 17:55:30.719350 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 17:55:30.719422 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:55:30.720338 (+ 916us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:55:30.720343 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:55:30.720367 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:55:30.720507 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:55:30.720513 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:55:30.722486 (+ 1973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:55:30.722490 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:55:30.725459 (+ 2969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:55:30.725466 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:55:30.728055 (+ 2589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:55:30.728059 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:55:30.729898 (+ 1839us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:55:30.729927 (+ 29us) client_negotiation.cc:769] Sending connection context
0929 17:55:30.729997 (+ 70us) client_negotiation.cc:240] Negotiation successful
0929 17:55:30.730070 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3288,"thread_start_us":113,"threads_started":1}
I20260929 17:55:30.731107 13669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:55:30.715907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60648 (local address 127.0.98.3:46287)
0929 17:55:30.719662 (+ 3755us) server_negotiation.cc:206] Beginning negotiation
0929 17:55:30.719670 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 17:55:30.719694 (+ 24us) server_negotiation.cc:407] Connection header received
0929 17:55:30.719771 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:55:30.719776 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:55:30.720016 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:55:30.720162 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:55:30.720633 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:55:30.722363 (+ 1730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:55:30.725586 (+ 3223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:55:30.726344 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:55:30.726449 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:55:30.730774 (+ 4325us) server_negotiation.cc:1035] Waiting for connection context
0929 17:55:30.730835 (+ 61us) server_negotiation.cc:299] Negotiation successful
0929 17:55:30.730962 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3643,"thread_start_us":94,"threads_started":1}
I20260929 17:55:30.732298 13307 tablet_service.cc:1467] Processing CreateTablet for tablet 78cd4fe5a3bb4c49a2493e31c36815a6 (DEFAULT_TABLE table=test-table [id=606d2da4314d477db56e6e4c26cfe0e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 17:55:30.733561 13307 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78cd4fe5a3bb4c49a2493e31c36815a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:55:30.734306 13557 tablet_service.cc:1467] Processing CreateTablet for tablet 78cd4fe5a3bb4c49a2493e31c36815a6 (DEFAULT_TABLE table=test-table [id=606d2da4314d477db56e6e4c26cfe0e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 17:55:30.735406 13557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78cd4fe5a3bb4c49a2493e31c36815a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:55:30.750130 13670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:55:30.716313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41657 (local address 127.0.0.1:34086)
0929 17:55:30.718997 (+ 2684us) negotiation.cc:107] Waiting for socket to connect
0929 17:55:30.719056 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0929 17:55:30.719185 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:55:30.745545 (+ 26360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:55:30.745549 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:55:30.745576 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:55:30.745702 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:55:30.745709 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:55:30.747364 (+ 1655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:55:30.747368 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:55:30.748436 (+ 1068us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:55:30.748445 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:55:30.749684 (+ 1239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:55:30.749689 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:55:30.749788 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:55:30.749819 (+ 31us) client_negotiation.cc:769] Sending connection context
0929 17:55:30.749881 (+ 62us) client_negotiation.cc:240] Negotiation successful
0929 17:55:30.749967 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2502,"thread_start_us":101,"threads_started":1}
I20260929 17:55:30.751125 13671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:55:30.716696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34086 (local address 127.0.98.2:41657)
0929 17:55:30.745032 (+ 28336us) server_negotiation.cc:206] Beginning negotiation
0929 17:55:30.745040 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 17:55:30.745068 (+ 28us) server_negotiation.cc:407] Connection header received
0929 17:55:30.745147 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:55:30.745153 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:55:30.745371 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:55:30.745872 (+ 501us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:55:30.746016 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:55:30.747162 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:55:30.748581 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:55:30.749548 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:55:30.750644 (+ 1096us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:55:30.750808 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0929 17:55:30.750879 (+ 71us) server_negotiation.cc:299] Negotiation successful
0929 17:55:30.750984 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28196,"thread_start_us":110,"threads_started":1}
I20260929 17:55:30.751801 13674 tablet_bootstrap.cc:492] T 78cd4fe5a3bb4c49a2493e31c36815a6 P 9ef29865b5a34644926e8db945ba9e68: Bootstrap starting.
I20260929 17:55:30.755646 13674 tablet_bootstrap.cc:654] T 78cd4fe5a3bb4c49a2493e31c36815a6 P 9ef29865b5a34644926e8db945ba9e68: Neither blocks nor log segments found. Creating new log.
I20260929 17:55:30.755381 13430 tablet_service.cc:1467] Processing CreateTablet for tablet 78cd4fe5a3bb4c49a2493e31c36815a6 (DEFAULT_TABLE table=test-table [id=606d2da4314d477db56e6e4c26cfe0e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 17:55:30.756615 13430 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78cd4fe5a3bb4c49a2493e31c36815a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:55:30.756788 13674 log.cc:824] T 78cd4fe5a3bb4c49a2493e31c36815a6 P 9ef29865b5a34644926e8db945ba9e68: Log is configured to *not* fsync() on all Append() calls
I20260929 17:55:30.759105 13674 tablet_bootstrap.cc:492] T 78cd4fe5a3bb4c49a2493e31c36815a6 P 9ef29865b5a34644926e8db945ba9e68: No bootstrap required, opened a new log
I20260929 17:55:30.759307 13674 ts_tablet_manager.cc:1397] T 78cd4fe5a3bb4c49a2493e31c36815a6 P 9ef29865b5a34644926e8db945ba9e68: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260929 17:55:30.764498 13674 raft_consensus.cc:348] T 78cd4fe5a3bb4c49a2493e31c36815a6 P 9ef29865b5a34644926e8db945ba9e68 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ef29865b5a34644926e8db945ba9e68" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46287 } } peers { permanent_uuid: "d793fe68b4284bf2b8ab8d9a4bfafaea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41657 } } peers { permanent_uuid: "982fb80772c64914a0c4e12631781c30" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40593 } }
I20260929 17:55:30.764992 13674 raft_consensus.cc:374] T 78cd4fe5a3bb4c49a2493e31c36815a6 P 9ef29865b5a34644926e8db945ba9e68 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:55:30.765192 13674 raft_consensus.cc:729] T 78cd4fe5a3bb4c49a2493e31c36815a6 P 9ef29865b5a34644926e8db945ba9e68 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ef29865b5a34644926e8db945ba9e68, State: Initialized, Role: FOLLOWER
I20260929 17:55:30.765861 13674 consensus_queue.cc:260] T 78cd4fe5a3bb4c49a2493e31c36815a6 P 9ef29865b5a34644926e8db945ba9e68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ef29865b5a34644926e8db945ba9e68" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46287 } } peers { permanent_uuid: "d793fe68b4284bf2b8ab8d9a4bfafaea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41657 } } peers { permanent_uuid: "982fb80772c64914a0c4e12631781c30" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40593 } }
I20260929 17:55:30.767253 13674 ts_tablet_manager.cc:1428] T 78cd4fe5a3bb4c49a2493e31c36815a6 P 9ef29865b5a34644926e8db945ba9e68: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260929 17:55:30.767712 13648 heartbeater.cc:503] Master 127.0.98.62:46201 was elected leader, sending a full tablet report...
I20260929 17:55:30.769228 13678 tablet_bootstrap.cc:492] T 78cd4fe5a3bb4c49a2493e31c36815a6 P d793fe68b4284bf2b8ab8d9a4bfafaea: Bootstrap starting.
I20260929 17:55:30.771740 13678 tablet_bootstrap.cc:654] T 78cd4fe5a3bb4c49a2493e31c36815a6 P d793fe68b4284bf2b8ab8d9a4bfafaea: Neither blocks nor log segments found. Creating new log.
I20260929 17:55:30.772689 13678 log.cc:824] T 78cd4fe5a3bb4c49a2493e31c36815a6 P d793fe68b4284bf2b8ab8d9a4bfafaea: Log is configured to *not* fsync() on all Append() calls
I20260929 17:55:30.773486 13676 tablet_bootstrap.cc:492] T 78cd4fe5a3bb4c49a2493e31c36815a6 P 982fb80772c64914a0c4e12631781c30: Bootstrap starting.
I20260929 17:55:30.775367 13676 tablet_bootstrap.cc:654] T 78cd4fe5a3bb4c49a2493e31c36815a6 P 982fb80772c64914a0c4e12631781c30: Neither blocks nor log segments found. Creating new log.
I20260929 17:55:30.776206 13676 log.cc:824] T 78cd4fe5a3bb4c49a2493e31c36815a6 P 982fb80772c64914a0c4e12631781c30: Log is configured to *not* fsync() on all Append() calls
I20260929 17:55:30.778908 13676 tablet_bootstrap.cc:492] T 78cd4fe5a3bb4c49a2493e31c36815a6 P 982fb80772c64914a0c4e12631781c30: No bootstrap required, opened a new log
I20260929 17:55:30.778934 13678 tablet_bootstrap.cc:492] T 78cd4fe5a3bb4c49a2493e31c36815a6 P d793fe68b4284bf2b8ab8d9a4bfafaea: No bootstrap required, opened a new log
I20260929 17:55:30.779114 13676 ts_tablet_manager.cc:1397] T 78cd4fe5a3bb4c49a2493e31c36815a6 P 982fb80772c64914a0c4e12631781c30: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20260929 17:55:30.779295 13678 ts_tablet_manager.cc:1397] T 78cd4fe5a3bb4c49a2493e31c36815a6 P d793fe68b4284bf2b8ab8d9a4bfafaea: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260929 17:55:30.783208 13676 raft_consensus.cc:348] T 78cd4fe5a3bb4c49a2493e31c36815a6 P 982fb80772c64914a0c4e12631781c30 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ef29865b5a34644926e8db945ba9e68" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46287 } } peers { permanent_uuid: "d793fe68b4284bf2b8ab8d9a4bfafaea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41657 } } peers { permanent_uuid: "982fb80772c64914a0c4e12631781c30" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40593 } }
I20260929 17:55:30.783512 13676 raft_consensus.cc:374] T 78cd4fe5a3bb4c49a2493e31c36815a6 P 982fb80772c64914a0c4e12631781c30 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:55:30.783567 13676 raft_consensus.cc:729] T 78cd4fe5a3bb4c49a2493e31c36815a6 P 982fb80772c64914a0c4e12631781c30 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 982fb80772c64914a0c4e12631781c30, State: Initialized, Role: FOLLOWER
I20260929 17:55:30.784094 13676 consensus_queue.cc:260] T 78cd4fe5a3bb4c49a2493e31c36815a6 P 982fb80772c64914a0c4e12631781c30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ef29865b5a34644926e8db945ba9e68" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46287 } } peers { permanent_uuid: "d793fe68b4284bf2b8ab8d9a4bfafaea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41657 } } peers { permanent_uuid: "982fb80772c64914a0c4e12631781c30" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40593 } }
I20260929 17:55:30.784140 13678 raft_consensus.cc:348] T 78cd4fe5a3bb4c49a2493e31c36815a6 P d793fe68b4284bf2b8ab8d9a4bfafaea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ef29865b5a34644926e8db945ba9e68" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46287 } } peers { permanent_uuid: "d793fe68b4284bf2b8ab8d9a4bfafaea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41657 } } peers { permanent_uuid: "982fb80772c64914a0c4e12631781c30" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40593 } }
I20260929 17:55:30.784521 13678 raft_consensus.cc:374] T 78cd4fe5a3bb4c49a2493e31c36815a6 P d793fe68b4284bf2b8ab8d9a4bfafaea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:55:30.784608 13678 raft_consensus.cc:729] T 78cd4fe5a3bb4c49a2493e31c36815a6 P d793fe68b4284bf2b8ab8d9a4bfafaea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d793fe68b4284bf2b8ab8d9a4bfafaea, State: Initialized, Role: FOLLOWER
I20260929 17:55:30.785167 13676 ts_tablet_manager.cc:1428] T 78cd4fe5a3bb4c49a2493e31c36815a6 P 982fb80772c64914a0c4e12631781c30: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260929 17:55:30.785122 13678 consensus_queue.cc:260] T 78cd4fe5a3bb4c49a2493e31c36815a6 P d793fe68b4284bf2b8ab8d9a4bfafaea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ef29865b5a34644926e8db945ba9e68" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46287 } } peers { permanent_uuid: "d793fe68b4284bf2b8ab8d9a4bfafaea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41657 } } peers { permanent_uuid: "982fb80772c64914a0c4e12631781c30" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40593 } }
I20260929 17:55:30.785516 13373 heartbeater.cc:503] Master 127.0.98.62:46201 was elected leader, sending a full tablet report...
I20260929 17:55:30.786262 13678 ts_tablet_manager.cc:1428] T 78cd4fe5a3bb4c49a2493e31c36815a6 P d793fe68b4284bf2b8ab8d9a4bfafaea: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260929 17:55:30.786643 13508 heartbeater.cc:503] Master 127.0.98.62:46201 was elected leader, sending a full tablet report...
I20260929 17:55:30.789431 13683 raft_consensus.cc:482] T 78cd4fe5a3bb4c49a2493e31c36815a6 P 982fb80772c64914a0c4e12631781c30 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 17:55:30.789575 13683 raft_consensus.cc:504] T 78cd4fe5a3bb4c49a2493e31c36815a6 P 982fb80772c64914a0c4e12631781c30 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ef29865b5a34644926e8db945ba9e68" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46287 } } peers { permanent_uuid: "d793fe68b4284bf2b8ab8d9a4bfafaea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41657 } } peers { permanent_uuid: "982fb80772c64914a0c4e12631781c30" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40593 } }
I20260929 17:55:30.791447 13683 leader_election.cc:288] T 78cd4fe5a3bb4c49a2493e31c36815a6 P 982fb80772c64914a0c4e12631781c30 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9ef29865b5a34644926e8db945ba9e68 (127.0.98.3:46287), d793fe68b4284bf2b8ab8d9a4bfafaea (127.0.98.2:41657)
I20260929 17:55:30.796602 13669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:55:30.791197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49363 (local address 127.0.98.3:46287)
0929 17:55:30.791309 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0929 17:55:30.791314 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:55:30.792282 (+ 968us) server_negotiation.cc:407] Connection header received
0929 17:55:30.792416 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:55:30.792419 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:55:30.792466 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:55:30.792543 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:55:30.792853 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:55:30.793661 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:55:30.795323 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:55:30.795955 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:55:30.796029 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:55:30.796090 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0929 17:55:30.796401 (+ 311us) server_negotiation.cc:299] Negotiation successful
0929 17:55:30.796476 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260929 17:55:30.797047 13685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:55:30.790871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46287 (local address 127.0.98.1:49363)
0929 17:55:30.792161 (+ 1290us) negotiation.cc:107] Waiting for socket to connect
0929 17:55:30.792177 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 17:55:30.792317 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:55:30.792562 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:55:30.792565 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:55:30.792584 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:55:30.792743 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:55:30.792750 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:55:30.793798 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:55:30.793801 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:55:30.795184 (+ 1383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:55:30.795194 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:55:30.796175 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:55:30.796178 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:55:30.796276 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:55:30.796297 (+ 21us) client_negotiation.cc:769] Sending connection context
0929 17:55:30.796788 (+ 491us) client_negotiation.cc:240] Negotiation successful
0929 17:55:30.796858 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1223,"mutex_wait_us":37,"thread_start_us":87,"threads_started":1}
I20260929 17:55:30.797641 13602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78cd4fe5a3bb4c49a2493e31c36815a6" candidate_uuid: "982fb80772c64914a0c4e12631781c30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ef29865b5a34644926e8db945ba9e68" is_pre_election: true
I20260929 17:55:30.797981 13671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:55:30.791748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51737 (local address 127.0.98.2:41657)
0929 17:55:30.791973 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0929 17:55:30.791978 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:55:30.793306 (+ 1328us) server_negotiation.cc:407] Connection header received
0929 17:55:30.793505 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:55:30.793509 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:55:30.793562 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:55:30.793615 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:55:30.794097 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:55:30.794799 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:55:30.796207 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:55:30.796777 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:55:30.796862 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:55:30.796924 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0929 17:55:30.797752 (+ 828us) server_negotiation.cc:299] Negotiation successful
0929 17:55:30.797859 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141}
I20260929 17:55:30.797966 13602 raft_consensus.cc:2454] T 78cd4fe5a3bb4c49a2493e31c36815a6 P 9ef29865b5a34644926e8db945ba9e68 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 982fb80772c64914a0c4e12631781c30 in term 0.
I20260929 17:55:30.798422 13263 leader_election.cc:302] T 78cd4fe5a3bb4c49a2493e31c36815a6 P 982fb80772c64914a0c4e12631781c30 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 982fb80772c64914a0c4e12631781c30, 9ef29865b5a34644926e8db945ba9e68; no voters:
I20260929 17:55:30.798756 13683 raft_consensus.cc:2781] T 78cd4fe5a3bb4c49a2493e31c36815a6 P 982fb80772c64914a0c4e12631781c30 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 17:55:30.798856 13683 raft_consensus.cc:482] T 78cd4fe5a3bb4c49a2493e31c36815a6 P 982fb80772c64914a0c4e12631781c30 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 17:55:30.798899 13683 raft_consensus.cc:3037] T 78cd4fe5a3bb4c49a2493e31c36815a6 P 982fb80772c64914a0c4e12631781c30 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:55:30.800127 13683 raft_consensus.cc:504] T 78cd4fe5a3bb4c49a2493e31c36815a6 P 982fb80772c64914a0c4e12631781c30 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ef29865b5a34644926e8db945ba9e68" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46287 } } peers { permanent_uuid: "d793fe68b4284bf2b8ab8d9a4bfafaea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41657 } } peers { permanent_uuid: "982fb80772c64914a0c4e12631781c30" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40593 } }
I20260929 17:55:30.800765 13602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78cd4fe5a3bb4c49a2493e31c36815a6" candidate_uuid: "982fb80772c64914a0c4e12631781c30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ef29865b5a34644926e8db945ba9e68"
I20260929 17:55:30.800905 13602 raft_consensus.cc:3037] T 78cd4fe5a3bb4c49a2493e31c36815a6 P 9ef29865b5a34644926e8db945ba9e68 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:55:30.802152 13602 raft_consensus.cc:2454] T 78cd4fe5a3bb4c49a2493e31c36815a6 P 9ef29865b5a34644926e8db945ba9e68 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 982fb80772c64914a0c4e12631781c30 in term 1.
I20260929 17:55:30.802453 13263 leader_election.cc:302] T 78cd4fe5a3bb4c49a2493e31c36815a6 P 982fb80772c64914a0c4e12631781c30 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 982fb80772c64914a0c4e12631781c30, 9ef29865b5a34644926e8db945ba9e68; no voters:
I20260929 17:55:30.802611 13684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:55:30.791675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41657 (local address 127.0.98.1:51737)
0929 17:55:30.793206 (+ 1531us) negotiation.cc:107] Waiting for socket to connect
0929 17:55:30.793221 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 17:55:30.793378 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:55:30.793763 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:55:30.793766 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:55:30.793785 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:55:30.793951 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:55:30.793960 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:55:30.794955 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:55:30.794958 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:55:30.796057 (+ 1099us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:55:30.796070 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:55:30.797432 (+ 1362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:55:30.797437 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:55:30.797540 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:55:30.797621 (+ 81us) client_negotiation.cc:769] Sending connection context
0929 17:55:30.798583 (+ 962us) client_negotiation.cc:240] Negotiation successful
0929 17:55:30.798719 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1464,"mutex_wait_us":4,"thread_start_us":101,"threads_started":1}
I20260929 17:55:30.803252 13448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78cd4fe5a3bb4c49a2493e31c36815a6" candidate_uuid: "982fb80772c64914a0c4e12631781c30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d793fe68b4284bf2b8ab8d9a4bfafaea" is_pre_election: true
I20260929 17:55:30.803401 13447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78cd4fe5a3bb4c49a2493e31c36815a6" candidate_uuid: "982fb80772c64914a0c4e12631781c30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d793fe68b4284bf2b8ab8d9a4bfafaea"
I20260929 17:55:30.803617 13448 raft_consensus.cc:2454] T 78cd4fe5a3bb4c49a2493e31c36815a6 P d793fe68b4284bf2b8ab8d9a4bfafaea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 982fb80772c64914a0c4e12631781c30 in term 0.
I20260929 17:55:30.802901 13681 raft_consensus.cc:2781] T 78cd4fe5a3bb4c49a2493e31c36815a6 P 982fb80772c64914a0c4e12631781c30 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 17:55:30.804304 13681 raft_consensus.cc:686] T 78cd4fe5a3bb4c49a2493e31c36815a6 P 982fb80772c64914a0c4e12631781c30 [term 1 LEADER]: Becoming Leader. State: Replica: 982fb80772c64914a0c4e12631781c30, State: Running, Role: LEADER
I20260929 17:55:30.803823 13683 leader_election.cc:288] T 78cd4fe5a3bb4c49a2493e31c36815a6 P 982fb80772c64914a0c4e12631781c30 [CANDIDATE]: Term 1 election: Requested vote from peers 9ef29865b5a34644926e8db945ba9e68 (127.0.98.3:46287), d793fe68b4284bf2b8ab8d9a4bfafaea (127.0.98.2:41657)
I20260929 17:55:30.804795 13681 consensus_queue.cc:237] T 78cd4fe5a3bb4c49a2493e31c36815a6 P 982fb80772c64914a0c4e12631781c30 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ef29865b5a34644926e8db945ba9e68" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46287 } } peers { permanent_uuid: "d793fe68b4284bf2b8ab8d9a4bfafaea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41657 } } peers { permanent_uuid: "982fb80772c64914a0c4e12631781c30" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40593 } }
I20260929 17:55:30.808525 13175 catalog_manager.cc:5495] T 78cd4fe5a3bb4c49a2493e31c36815a6 P 982fb80772c64914a0c4e12631781c30 reported cstate change: term changed from 0 to 1, leader changed from <none> to 982fb80772c64914a0c4e12631781c30 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "982fb80772c64914a0c4e12631781c30" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ef29865b5a34644926e8db945ba9e68" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46287 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d793fe68b4284bf2b8ab8d9a4bfafaea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "982fb80772c64914a0c4e12631781c30" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40593 } health_report { overall_health: HEALTHY } } }
I20260929 17:55:30.841459 13673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:55:30.836698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51208 (local address 127.0.98.1:40593)
0929 17:55:30.836853 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0929 17:55:30.836858 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:55:30.836907 (+ 49us) server_negotiation.cc:407] Connection header received
0929 17:55:30.837032 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:55:30.837035 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:55:30.837128 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:55:30.837254 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 17:55:30.837500 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:55:30.838416 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:55:30.839627 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:55:30.839976 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:55:30.840081 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:55:30.840406 (+ 325us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 17:55:30.841056 (+ 650us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 17:55:30.841179 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0929 17:55:30.841217 (+ 38us) server_negotiation.cc:299] Negotiation successful
0929 17:55:30.841287 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260929 17:55:30.848435 13447 raft_consensus.cc:3037] T 78cd4fe5a3bb4c49a2493e31c36815a6 P d793fe68b4284bf2b8ab8d9a4bfafaea [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:55:30.848634 13602 raft_consensus.cc:1261] T 78cd4fe5a3bb4c49a2493e31c36815a6 P 9ef29865b5a34644926e8db945ba9e68 [term 1 FOLLOWER]: Refusing update from remote peer 982fb80772c64914a0c4e12631781c30: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 17:55:30.849479 13681 consensus_queue.cc:1035] T 78cd4fe5a3bb4c49a2493e31c36815a6 P 982fb80772c64914a0c4e12631781c30 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ef29865b5a34644926e8db945ba9e68" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46287 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 17:55:30.850039 13447 raft_consensus.cc:1261] T 78cd4fe5a3bb4c49a2493e31c36815a6 P d793fe68b4284bf2b8ab8d9a4bfafaea [term 1 FOLLOWER]: Refusing update from remote peer 982fb80772c64914a0c4e12631781c30: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 17:55:30.850693 13681 consensus_queue.cc:1035] T 78cd4fe5a3bb4c49a2493e31c36815a6 P 982fb80772c64914a0c4e12631781c30 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d793fe68b4284bf2b8ab8d9a4bfafaea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 17:55:30.860936 13689 mvcc.cc:205] Tried to move back new op lower bound from 7334725758347743232 to 7334725758180413440. Current Snapshot: MvccSnapshot[applied={T|T < 7334725758347743232}]
I20260929 17:55:30.864657 13691 mvcc.cc:205] Tried to move back new op lower bound from 7334725758347743232 to 7334725758180413440. Current Snapshot: MvccSnapshot[applied={T|T < 7334725758347743232}]
I20260929 17:55:30.867285 13690 mvcc.cc:205] Tried to move back new op lower bound from 7334725758347743232 to 7334725758180413440. Current Snapshot: MvccSnapshot[applied={T|T < 7334725758347743232}]
I20260929 17:55:30.874810 13175 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33716:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 17:55:30.875022 13175 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33716:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 17:55:30.878042 13175 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c1c7f1bbb54d4f84afc44bb16d650329: Sending DeleteTablet for 3 replicas of tablet 78cd4fe5a3bb4c49a2493e31c36815a6
I20260929 17:55:30.878912 13307 tablet_service.cc:1514] Processing DeleteTablet for tablet 78cd4fe5a3bb4c49a2493e31c36815a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 17:55:30 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51194
I20260929 17:55:30.879103 13430 tablet_service.cc:1514] Processing DeleteTablet for tablet 78cd4fe5a3bb4c49a2493e31c36815a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 17:55:30 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34086
I20260929 17:55:30.879558 13699 tablet_replica.cc:330] T 78cd4fe5a3bb4c49a2493e31c36815a6 P d793fe68b4284bf2b8ab8d9a4bfafaea: stopping tablet replica
I20260929 17:55:30.879596 13557 tablet_service.cc:1514] Processing DeleteTablet for tablet 78cd4fe5a3bb4c49a2493e31c36815a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 17:55:30 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60648
I20260929 17:55:30.879662 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskup3Soe/build/debug/bin/kudu with pid 13213
I20260929 17:55:30.880731 13699 raft_consensus.cc:2229] T 78cd4fe5a3bb4c49a2493e31c36815a6 P d793fe68b4284bf2b8ab8d9a4bfafaea [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 17:55:30.881152 13699 raft_consensus.cc:2258] T 78cd4fe5a3bb4c49a2493e31c36815a6 P d793fe68b4284bf2b8ab8d9a4bfafaea [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 17:55:30.882900 13699 ts_tablet_manager.cc:1905] T 78cd4fe5a3bb4c49a2493e31c36815a6 P d793fe68b4284bf2b8ab8d9a4bfafaea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 17:55:30.883783 13700 tablet_replica.cc:330] T 78cd4fe5a3bb4c49a2493e31c36815a6 P 9ef29865b5a34644926e8db945ba9e68: stopping tablet replica
I20260929 17:55:30.885836 13699 ts_tablet_manager.cc:1918] T 78cd4fe5a3bb4c49a2493e31c36815a6 P d793fe68b4284bf2b8ab8d9a4bfafaea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 17:55:30.885954 13699 log.cc:1196] T 78cd4fe5a3bb4c49a2493e31c36815a6 P d793fe68b4284bf2b8ab8d9a4bfafaea: Deleting WAL directory at /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704375975685-392-0/minicluster-data/ts-1/wal/wals/78cd4fe5a3bb4c49a2493e31c36815a6
I20260929 17:55:30.886272 13699 ts_tablet_manager.cc:1939] T 78cd4fe5a3bb4c49a2493e31c36815a6 P d793fe68b4284bf2b8ab8d9a4bfafaea: Deleting consensus metadata
I20260929 17:55:30.887723 13160 catalog_manager.cc:4841] TS d793fe68b4284bf2b8ab8d9a4bfafaea (127.0.98.2:41657): tablet 78cd4fe5a3bb4c49a2493e31c36815a6 (table test-table [id=606d2da4314d477db56e6e4c26cfe0e0]) successfully deleted
I20260929 17:55:30.887929 13700 raft_consensus.cc:2229] T 78cd4fe5a3bb4c49a2493e31c36815a6 P 9ef29865b5a34644926e8db945ba9e68 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 17:55:30.888278 13700 raft_consensus.cc:2258] T 78cd4fe5a3bb4c49a2493e31c36815a6 P 9ef29865b5a34644926e8db945ba9e68 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260929 17:55:30.888689 13159 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:40593 (error 108)
I20260929 17:55:30.889555 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskup3Soe/build/debug/bin/kudu with pid 13377
I20260929 17:55:30.889750 13670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:55:30.889496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40593 (local address 127.0.0.1:51216)
0929 17:55:30.889606 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0929 17:55:30.889671 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260929 17:55:30.890885 13159 catalog_manager.cc:4584] TS 982fb80772c64914a0c4e12631781c30 (127.0.98.1:40593): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 78cd4fe5a3bb4c49a2493e31c36815a6: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40593: connect: Connection refused (error 111)
I20260929 17:55:30.891768 13700 ts_tablet_manager.cc:1905] T 78cd4fe5a3bb4c49a2493e31c36815a6 P 9ef29865b5a34644926e8db945ba9e68: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 17:55:30.894284 13700 ts_tablet_manager.cc:1918] T 78cd4fe5a3bb4c49a2493e31c36815a6 P 9ef29865b5a34644926e8db945ba9e68: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 17:55:30.894412 13700 log.cc:1196] T 78cd4fe5a3bb4c49a2493e31c36815a6 P 9ef29865b5a34644926e8db945ba9e68: Deleting WAL directory at /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790704375975685-392-0/minicluster-data/ts-2/wal/wals/78cd4fe5a3bb4c49a2493e31c36815a6
I20260929 17:55:30.894748 13700 ts_tablet_manager.cc:1939] T 78cd4fe5a3bb4c49a2493e31c36815a6 P 9ef29865b5a34644926e8db945ba9e68: Deleting consensus metadata
I20260929 17:55:30.896276 13162 catalog_manager.cc:4841] TS 9ef29865b5a34644926e8db945ba9e68 (127.0.98.3:46287): tablet 78cd4fe5a3bb4c49a2493e31c36815a6 (table test-table [id=606d2da4314d477db56e6e4c26cfe0e0]) successfully deleted
I20260929 17:55:30.897075 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskup3Soe/build/debug/bin/kudu with pid 13511
I20260929 17:55:30.905491 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskup3Soe/build/debug/bin/kudu with pid 13142
I20260929 17:55:30.913369 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260929 17:55:31.248127 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260929 17:55:31.248230 392 mini_ranger.cc:66] Stopping Ranger...
I20260929 17:55:31.582634 392 mini_ranger.cc:68] Stopped Ranger
2026-09-29 17:55:31.582 UTC [12209] LOG: received smart shutdown request
2026-09-29 17:55:31.586 UTC [12209] LOG: background worker "logical replication launcher" (PID 12218) exited with exit code 1
2026-09-29 17:55:31.586 UTC [12213] LOG: shutting down
2026-09-29 17:55:31.741 UTC [12209] LOG: database system is shut down
2026-09-29T17:55:31Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (47288 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790704375975685-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 17:55:31 dist-test-slave-53nh krb5kdc[13710](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 17:55:31 dist-test-slave-53nh krb5kdc[13710](info): set up 2 sockets
Sep 29 17:55:31 dist-test-slave-53nh krb5kdc[13710](info): commencing operation
krb5kdc: starting...
W20260929 17:55:32.792788 392 mini_kdc.cc:121] Time spent starting KDC: real 0.940s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790704375975685-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790704375975685-392-0/krb5kdc/test-user.keytab.
Sep 29 17:55:32 dist-test-slave-53nh krb5kdc[13710](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704532, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T17:55:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T17:55:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790704375975685-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790704375975685-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 17:55:32.904198 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskup3Soe/build/debug/bin/kudu
/tmp/dist-test-taskup3Soe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790704375975685-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790704375975685-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790704375975685-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790704375975685-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39429
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790704375975685-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39429,127.0.98.61:36315,127.0.98.60:33313
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790704375975685-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790704375975685-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790704375975685-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:55:33.013628 13723 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:55:33.013921 13723 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:55:33.013965 13723 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:55:33.017983 13723 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 17:55:33.018061 13723 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:55:33.018083 13723 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:55:33.018105 13723 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 17:55:33.018139 13723 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 17:55:33.023787 13723 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790704375975685-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790704375975685-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39429,127.0.98.61:36315,127.0.98.60:33313
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790704375975685-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39429
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790704375975685-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13723
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790704375975685-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:55:33.025202 13723 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:55:33.026506 13723 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:55:33.035341 13731 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:55:33.035341 13729 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:55:33.035547 13723 server_base.cc:1034] running on GCE node
W20260929 17:55:33.035391 13728 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:55:33.036101 13723 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:55:33.037212 13723 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:55:33.038378 13723 hybrid_clock.cc:648] HybridClock initialized: now 1790704533038344 us; error 44 us; skew 500 ppm
Sep 29 17:55:33 dist-test-slave-53nh krb5kdc[13710](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704533, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:55:33.044610 13723 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 17:55:33.045746 13723 webserver.cc:460] Webserver started at http://127.0.98.62:43733/ using document root <none> and password file <none>
I20260929 17:55:33.046286 13723 fs_manager.cc:359] Metadata directory not provided
I20260929 17:55:33.046351 13723 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:55:33.046545 13723 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:55:33.048548 13723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790704375975685-392-0/minicluster-data/master-0/data/instance:
uuid: "c5142b27fb864cd993938fa340d3bc9a"
format_stamp: "Formatted at 2026-09-29 17:55:33 on dist-test-slave-53nh"
server_key: "7ba061b5164a5b471c7ff205159e0282"
server_key_iv: "55887d6e6137432001192c56efde9be4"
server_key_version: "encryptionkey@0"
I20260929 17:55:33.049119 13723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790704375975685-392-0/minicluster-data/master-0/wal/instance:
uuid: "c5142b27fb864cd993938fa340d3bc9a"
format_stamp: "Formatted at 2026-09-29 17:55:33 on dist-test-slave-53nh"
server_key: "7ba061b5164a5b471c7ff205159e0282"
server_key_iv: "55887d6e6137432001192c56efde9be4"
server_key_version: "encryptionkey@0"
I20260929 17:55:33.052783 13723 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260929 17:55:33.055279 13738 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:55:33.056466 13723 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 17:55:33.056586 13723 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790704375975685-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790704375975685-392-0/minicluster-data/master-0/wal
uuid: "c5142b27fb864cd993938fa340d3bc9a"
format_stamp: "Formatted at 2026-09-29 17:55:33 on dist-test-slave-53nh"
server_key: "7ba061b5164a5b471c7ff205159e0282"
server_key_iv: "55887d6e6137432001192c56efde9be4"
server_key_version: "encryptionkey@0"
I20260929 17:55:33.056680 13723 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790704375975685-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790704375975685-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790704375975685-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:55:33.065620 13723 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:55:33.068248 13723 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:55:33.068411 13723 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:55:33.075001 13723 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39429
I20260929 17:55:33.075033 13790 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39429 every 8 connection(s)
I20260929 17:55:33.075897 13723 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790704375975685-392-0/minicluster-data/master-0/data/info.pb
I20260929 17:55:33.078541 13791 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:55:33.079664 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskup3Soe/build/debug/bin/kudu as pid 13723
I20260929 17:55:33.079777 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790704375975685-392-0/minicluster-data/master-0/wal/instance
I20260929 17:55:33.080063 392 external_mini_cluster.cc:1428] Setting key 518a4b9f3c60716d3655d82f3fb428a8
I20260929 17:55:33.082758 13791 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39429 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Sep 29 17:55:33 dist-test-slave-53nh krb5kdc[13710](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704533, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 17:55:33.097304 13794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:55:33.085140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50614 (local address 127.0.98.62:39429)
0929 17:55:33.085689 (+ 549us) server_negotiation.cc:206] Beginning negotiation
0929 17:55:33.085706 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 17:55:33.085872 (+ 166us) server_negotiation.cc:407] Connection header received
0929 17:55:33.086977 (+ 1105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:55:33.086982 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:55:33.087152 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:55:33.087323 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:55:33.088153 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:55:33.088988 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:55:33.089993 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:55:33.090499 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:55:33.090645 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:55:33.092896 (+ 2251us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:55:33.092901 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:55:33.092911 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:55:33.092930 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:55:33.094493 (+ 1563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:55:33.095010 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:55:33.095014 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:55:33.095015 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:55:33.095109 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:55:33.095409 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:55:33.095410 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:55:33.095412 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:55:33.095745 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:55:33.095841 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0929 17:55:33.096256 (+ 415us) server_negotiation.cc:299] Negotiation successful
0929 17:55:33.096480 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":416,"thread_start_us":128,"threads_started":1}
I20260929 17:55:33.097302 13795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:55:33.085141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39429 (local address 127.0.0.1:50614)
0929 17:55:33.085741 (+ 600us) negotiation.cc:107] Waiting for socket to connect
0929 17:55:33.085780 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 17:55:33.086670 (+ 890us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:55:33.087329 (+ 659us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:55:33.087334 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:55:33.087743 (+ 409us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:55:33.088019 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:55:33.088032 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:55:33.089117 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:55:33.089120 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:55:33.089867 (+ 747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:55:33.089876 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:55:33.090661 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:55:33.090664 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:55:33.090762 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:55:33.091287 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:55:33.091303 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:55:33.092741 (+ 1438us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:55:33.094600 (+ 1859us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:55:33.094610 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:55:33.094621 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:55:33.094914 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:55:33.095187 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:55:33.095191 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:55:33.095193 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:55:33.095330 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:55:33.095852 (+ 522us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:55:33.095858 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:55:33.096027 (+ 169us) client_negotiation.cc:769] Sending connection context
0929 17:55:33.096215 (+ 188us) client_negotiation.cc:240] Negotiation successful
0929 17:55:33.096547 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":414,"mutex_wait_us":47,"spinlock_wait_cycles":16640,"thread_start_us":156,"threads_started":1}
I20260929 17:55:33.099479 13791 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36315 } has no permanent_uuid. Determining permanent_uuid...
I20260929 17:55:33.100154 13795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:55:33.099888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36315 (local address 127.0.0.1:38248)
0929 17:55:33.099998 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0929 17:55:33.100095 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260929 17:55:33.100330 13742 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36315: connect: Connection refused (error 111)
I20260929 17:55:33.101195 13795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:55:33.100920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36315 (local address 127.0.0.1:38260)
0929 17:55:33.101068 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0929 17:55:33.101136 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260929 17:55:33.101313 13791 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:36315: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36315: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790704375975685-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790704375975685-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260929 17:55:33.105333 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskup3Soe/build/debug/bin/kudu
/tmp/dist-test-taskup3Soe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790704375975685-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790704375975685-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790704375975685-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790704375975685-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:36315
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790704375975685-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39429,127.0.98.61:36315,127.0.98.60:33313
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790704375975685-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790704375975685-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790704375975685-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 17:55:33.150544 13791 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36315 } attempt: 1
I20260929 17:55:33.151386 13795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:55:33.151098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36315 (local address 127.0.0.1:38262)
0929 17:55:33.151245 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0929 17:55:33.151308 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260929 17:55:33.152052 13795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:55:33.151848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36315 (local address 127.0.0.1:38266)
0929 17:55:33.151946 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0929 17:55:33.151993 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260929 17:55:33.152165 13791 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:36315: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36315: connect: Connection refused (error 111)
W20260929 17:55:33.211987 13798 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:55:33.212276 13798 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:55:33.212318 13798 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:55:33.216396 13798 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 17:55:33.216473 13798 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:55:33.216491 13798 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:55:33.216507 13798 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 17:55:33.216524 13798 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 17:55:33.220304 13791 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36315 } attempt: 2
I20260929 17:55:33.221127 13795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:55:33.220800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36315 (local address 127.0.0.1:38280)
0929 17:55:33.220944 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0929 17:55:33.221045 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260929 17:55:33.221813 13795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:55:33.221635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36315 (local address 127.0.0.1:38290)
0929 17:55:33.221718 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0929 17:55:33.221761 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260929 17:55:33.221935 13791 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:36315: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36315: connect: Connection refused (error 111)
I20260929 17:55:33.222184 13798 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790704375975685-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790704375975685-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39429,127.0.98.61:36315,127.0.98.60:33313
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790704375975685-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:36315
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790704375975685-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13798
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790704375975685-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:55:33.223246 13798 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:55:33.224280 13798 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:55:33.232613 13804 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:55:33.232753 13798 server_base.cc:1034] running on GCE node
W20260929 17:55:33.232666 13803 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:55:33.232661 13806 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:55:33.233479 13798 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:55:33.234789 13798 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:55:33.235941 13798 hybrid_clock.cc:648] HybridClock initialized: now 1790704533235916 us; error 44 us; skew 500 ppm
Sep 29 17:55:33 dist-test-slave-53nh krb5kdc[13710](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704533, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:55:33.238886 13798 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260929 17:55:33.239872 13798 webserver.cc:460] Webserver started at http://127.0.98.61:38685/ using document root <none> and password file <none>
I20260929 17:55:33.240427 13798 fs_manager.cc:359] Metadata directory not provided
I20260929 17:55:33.240496 13798 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:55:33.240702 13798 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:55:33.242835 13798 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790704375975685-392-0/minicluster-data/master-1/data/instance:
uuid: "f39aedce090a401cb3a32682669b1876"
format_stamp: "Formatted at 2026-09-29 17:55:33 on dist-test-slave-53nh"
server_key: "af322413ec42dc8fa91bfc19482de95b"
server_key_iv: "c51a573fa75cd013275c96fe95343bf8"
server_key_version: "encryptionkey@0"
I20260929 17:55:33.243394 13798 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790704375975685-392-0/minicluster-data/master-1/wal/instance:
uuid: "f39aedce090a401cb3a32682669b1876"
format_stamp: "Formatted at 2026-09-29 17:55:33 on dist-test-slave-53nh"
server_key: "af322413ec42dc8fa91bfc19482de95b"
server_key_iv: "c51a573fa75cd013275c96fe95343bf8"
server_key_version: "encryptionkey@0"
I20260929 17:55:33.247381 13798 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 17:55:33.250030 13813 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:55:33.251317 13798 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 17:55:33.251436 13798 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790704375975685-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790704375975685-392-0/minicluster-data/master-1/wal
uuid: "f39aedce090a401cb3a32682669b1876"
format_stamp: "Formatted at 2026-09-29 17:55:33 on dist-test-slave-53nh"
server_key: "af322413ec42dc8fa91bfc19482de95b"
server_key_iv: "c51a573fa75cd013275c96fe95343bf8"
server_key_version: "encryptionkey@0"
I20260929 17:55:33.251547 13798 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790704375975685-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790704375975685-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790704375975685-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:55:33.266937 13798 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:55:33.270426 13798 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:55:33.270619 13798 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:55:33.277244 13798 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:36315
I20260929 17:55:33.277266 13865 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:36315 every 8 connection(s)
I20260929 17:55:33.278165 13798 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790704375975685-392-0/minicluster-data/master-1/data/info.pb
I20260929 17:55:33.280812 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskup3Soe/build/debug/bin/kudu as pid 13798
I20260929 17:55:33.280925 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790704375975685-392-0/minicluster-data/master-1/wal/instance
I20260929 17:55:33.280941 13866 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:55:33.281260 392 external_mini_cluster.cc:1428] Setting key 85180e39c668f6a58331d6336207c371
I20260929 17:55:33.285107 13866 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39429 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Sep 29 17:55:33 dist-test-slave-53nh krb5kdc[13710](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704533, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 17:55:33.299075 13794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:55:33.286987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50618 (local address 127.0.98.62:39429)
0929 17:55:33.287150 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0929 17:55:33.287155 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:55:33.288001 (+ 846us) server_negotiation.cc:407] Connection header received
0929 17:55:33.288976 (+ 975us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:55:33.288979 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:55:33.289065 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:55:33.289152 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:55:33.290451 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:55:33.291144 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:55:33.292658 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:55:33.293022 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:55:33.293125 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:55:33.295719 (+ 2594us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:55:33.295722 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:55:33.295724 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:55:33.295749 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:55:33.297074 (+ 1325us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:55:33.297641 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:55:33.297643 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:55:33.297644 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:55:33.297714 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:55:33.298053 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:55:33.298055 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:55:33.298056 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:55:33.298221 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:55:33.298284 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0929 17:55:33.298809 (+ 525us) server_negotiation.cc:299] Negotiation successful
0929 17:55:33.298905 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 17:55:33.299835 13869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:55:33.287340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39429 (local address 127.0.0.1:50618)
0929 17:55:33.287831 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0929 17:55:33.287884 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0929 17:55:33.288771 (+ 887us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:55:33.289328 (+ 557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:55:33.289335 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:55:33.289685 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:55:33.290299 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:55:33.290315 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:55:33.291274 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:55:33.291277 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:55:33.292545 (+ 1268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:55:33.292555 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:55:33.293155 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:55:33.293158 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:55:33.293306 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:55:33.293853 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:55:33.293870 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:55:33.295547 (+ 1677us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:55:33.297178 (+ 1631us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:55:33.297184 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:55:33.297200 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:55:33.297553 (+ 353us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:55:33.297799 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:55:33.297802 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:55:33.297804 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:55:33.297979 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:55:33.298325 (+ 346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:55:33.298332 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:55:33.298608 (+ 276us) client_negotiation.cc:769] Sending connection context
0929 17:55:33.298764 (+ 156us) client_negotiation.cc:240] Negotiation successful
0929 17:55:33.299036 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":129,"threads_started":1}
I20260929 17:55:33.301133 13866 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36315 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 17:55:33 dist-test-slave-53nh krb5kdc[13710](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704533, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790704375975685-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790704375975685-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260929 17:55:33.307719 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskup3Soe/build/debug/bin/kudu
/tmp/dist-test-taskup3Soe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790704375975685-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790704375975685-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790704375975685-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790704375975685-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:33313
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790704375975685-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39429,127.0.98.61:36315,127.0.98.60:33313
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790704375975685-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790704375975685-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790704375975685-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 17:55:33.309912 13869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:55:33.301594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36315 (local address 127.0.0.1:38306)
0929 17:55:33.301704 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0929 17:55:33.301718 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 17:55:33.301789 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:55:33.302375 (+ 586us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:55:33.302378 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:55:33.302505 (+ 127us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:55:33.302609 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:55:33.302616 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:55:33.303637 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:55:33.303640 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:55:33.304247 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:55:33.304256 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:55:33.304810 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:55:33.304813 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:55:33.304885 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:55:33.305390 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:55:33.305401 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:55:33.306530 (+ 1129us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:55:33.308428 (+ 1898us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:55:33.308430 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:55:33.308432 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:55:33.308619 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:55:33.308880 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:55:33.308883 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:55:33.308885 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:55:33.308971 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:55:33.309398 (+ 427us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:55:33.309401 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:55:33.309511 (+ 110us) client_negotiation.cc:769] Sending connection context
0929 17:55:33.309608 (+ 97us) client_negotiation.cc:240] Negotiation successful
0929 17:55:33.309727 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260929 17:55:33.309933 13871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:55:33.301726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38306 (local address 127.0.98.61:36315)
0929 17:55:33.302045 (+ 319us) server_negotiation.cc:206] Beginning negotiation
0929 17:55:33.302056 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 17:55:33.302078 (+ 22us) server_negotiation.cc:407] Connection header received
0929 17:55:33.302115 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:55:33.302119 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:55:33.302266 (+ 147us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:55:33.302369 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:55:33.302699 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:55:33.303520 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:55:33.304342 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:55:33.304703 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:55:33.304807 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:55:33.306644 (+ 1837us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:55:33.306654 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:55:33.306668 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:55:33.306690 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:55:33.308332 (+ 1642us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:55:33.308709 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:55:33.308715 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:55:33.308717 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:55:33.308799 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:55:33.309079 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:55:33.309082 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:55:33.309084 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:55:33.309315 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:55:33.309415 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0929 17:55:33.309673 (+ 258us) server_negotiation.cc:299] Negotiation successful
0929 17:55:33.309785 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":210,"thread_start_us":134,"threads_started":1}
I20260929 17:55:33.311805 13866 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33313 } has no permanent_uuid. Determining permanent_uuid...
I20260929 17:55:33.312439 13869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:55:33.312227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33313 (local address 127.0.0.1:41698)
0929 17:55:33.312317 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0929 17:55:33.312388 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260929 17:55:33.312671 13814 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33313: connect: Connection refused (error 111)
I20260929 17:55:33.313086 13791 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36315 } attempt: 3
I20260929 17:55:33.313488 13869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:55:33.313269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33313 (local address 127.0.0.1:41708)
0929 17:55:33.313390 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0929 17:55:33.313432 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260929 17:55:33.314275 13866 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:33313: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33313: connect: Connection refused (error 111)
Sep 29 17:55:33 dist-test-slave-53nh krb5kdc[13710](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704533, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260929 17:55:33.322237 13795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:55:33.313486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36315 (local address 127.0.0.1:38310)
0929 17:55:33.313744 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0929 17:55:33.313760 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 17:55:33.313847 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:55:33.314224 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:55:33.314227 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:55:33.314365 (+ 138us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:55:33.314827 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:55:33.314836 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:55:33.315808 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:55:33.315811 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:55:33.316365 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:55:33.316373 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:55:33.317513 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:55:33.317516 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:55:33.317600 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:55:33.318082 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:55:33.318093 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:55:33.319214 (+ 1121us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:55:33.320916 (+ 1702us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:55:33.320919 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:55:33.320920 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:55:33.321161 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:55:33.321413 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:55:33.321415 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:55:33.321418 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:55:33.321506 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:55:33.321844 (+ 338us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:55:33.321845 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:55:33.321926 (+ 81us) client_negotiation.cc:769] Sending connection context
0929 17:55:33.322011 (+ 85us) client_negotiation.cc:240] Negotiation successful
0929 17:55:33.322114 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147}
I20260929 17:55:33.322317 13871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:55:33.313539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38310 (local address 127.0.98.61:36315)
0929 17:55:33.314017 (+ 478us) server_negotiation.cc:206] Beginning negotiation
0929 17:55:33.314021 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:55:33.314033 (+ 12us) server_negotiation.cc:407] Connection header received
0929 17:55:33.314065 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:55:33.314068 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:55:33.314110 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:55:33.314187 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:55:33.314943 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:55:33.315719 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:55:33.316474 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:55:33.317381 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:55:33.317512 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:55:33.319335 (+ 1823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:55:33.319339 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:55:33.319342 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:55:33.319367 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:55:33.320822 (+ 1455us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:55:33.321239 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:55:33.321242 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:55:33.321244 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:55:33.321322 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:55:33.321576 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:55:33.321579 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:55:33.321581 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:55:33.321768 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:55:33.321846 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0929 17:55:33.322067 (+ 221us) server_negotiation.cc:299] Negotiation successful
0929 17:55:33.322174 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":406}
I20260929 17:55:33.323256 13791 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33313 } has no permanent_uuid. Determining permanent_uuid...
I20260929 17:55:33.323927 13795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:55:33.323703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33313 (local address 127.0.0.1:41718)
0929 17:55:33.323815 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0929 17:55:33.323866 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260929 17:55:33.324572 13795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:55:33.324362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33313 (local address 127.0.0.1:41730)
0929 17:55:33.324485 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0929 17:55:33.324524 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
W20260929 17:55:33.324698 13791 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:33313: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33313: connect: Connection refused (error 111)
I20260929 17:55:33.355856 13791 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33313 } attempt: 1
I20260929 17:55:33.356669 13795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:55:33.356386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33313 (local address 127.0.0.1:41734)
0929 17:55:33.356536 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0929 17:55:33.356592 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260929 17:55:33.357328 13795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:55:33.357133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33313 (local address 127.0.0.1:41744)
0929 17:55:33.357231 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0929 17:55:33.357273 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260929 17:55:33.357476 13791 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:33313: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33313: connect: Connection refused (error 111)
I20260929 17:55:33.363544 13866 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33313 } attempt: 1
I20260929 17:55:33.364213 13869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:55:33.363979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33313 (local address 127.0.0.1:41748)
0929 17:55:33.364098 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0929 17:55:33.364147 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260929 17:55:33.364858 13869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:55:33.364677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33313 (local address 127.0.0.1:41764)
0929 17:55:33.364763 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0929 17:55:33.364805 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260929 17:55:33.364964 13866 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:33313: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33313: connect: Connection refused (error 111)
W20260929 17:55:33.417383 13872 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:55:33.417672 13872 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:55:33.417713 13872 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:55:33.421747 13872 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 17:55:33.421821 13872 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:55:33.421840 13872 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:55:33.421871 13872 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 17:55:33.421898 13872 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 17:55:33.427225 13872 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790704375975685-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790704375975685-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39429,127.0.98.61:36315,127.0.98.60:33313
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790704375975685-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:33313
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790704375975685-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13872
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790704375975685-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:55:33.428289 13872 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:55:33.429518 13872 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 17:55:33.432663 13791 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33313 } attempt: 2
I20260929 17:55:33.433115 13866 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33313 } attempt: 2
I20260929 17:55:33.433568 13795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:55:33.433225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33313 (local address 127.0.0.1:41772)
0929 17:55:33.433403 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0929 17:55:33.433470 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260929 17:55:33.433810 13869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:55:33.433509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33313 (local address 127.0.0.1:41778)
0929 17:55:33.433686 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0929 17:55:33.433739 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109}
I20260929 17:55:33.434399 13869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:55:33.434226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33313 (local address 127.0.0.1:41804)
0929 17:55:33.434309 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0929 17:55:33.434352 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260929 17:55:33.434401 13795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:55:33.434145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33313 (local address 127.0.0.1:41794)
0929 17:55:33.434282 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0929 17:55:33.434335 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260929 17:55:33.434558 13791 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:33313: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33313: connect: Connection refused (error 111)
W20260929 17:55:33.434686 13866 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:33313: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33313: connect: Connection refused (error 111)
W20260929 17:55:33.437904 13881 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:55:33.437893 13879 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:55:33.437892 13878 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:55:33.438148 13872 server_base.cc:1034] running on GCE node
I20260929 17:55:33.438848 13872 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:55:33.439991 13872 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:55:33.441145 13872 hybrid_clock.cc:648] HybridClock initialized: now 1790704533441114 us; error 41 us; skew 500 ppm
Sep 29 17:55:33 dist-test-slave-53nh krb5kdc[13710](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704533, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:55:33.443873 13872 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260929 17:55:33.444712 13872 webserver.cc:460] Webserver started at http://127.0.98.60:34529/ using document root <none> and password file <none>
I20260929 17:55:33.445262 13872 fs_manager.cc:359] Metadata directory not provided
I20260929 17:55:33.445322 13872 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:55:33.445485 13872 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:55:33.447420 13872 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790704375975685-392-0/minicluster-data/master-2/data/instance:
uuid: "3e760625c2e54fe4bc2d405a34eb439a"
format_stamp: "Formatted at 2026-09-29 17:55:33 on dist-test-slave-53nh"
server_key: "f758378d47f588333ad7c35839673562"
server_key_iv: "219c09497f24cf39065a3b33a44b3597"
server_key_version: "encryptionkey@0"
I20260929 17:55:33.447906 13872 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790704375975685-392-0/minicluster-data/master-2/wal/instance:
uuid: "3e760625c2e54fe4bc2d405a34eb439a"
format_stamp: "Formatted at 2026-09-29 17:55:33 on dist-test-slave-53nh"
server_key: "f758378d47f588333ad7c35839673562"
server_key_iv: "219c09497f24cf39065a3b33a44b3597"
server_key_version: "encryptionkey@0"
I20260929 17:55:33.451460 13872 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260929 17:55:33.453784 13888 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:55:33.455008 13872 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 17:55:33.455133 13872 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790704375975685-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790704375975685-392-0/minicluster-data/master-2/wal
uuid: "3e760625c2e54fe4bc2d405a34eb439a"
format_stamp: "Formatted at 2026-09-29 17:55:33 on dist-test-slave-53nh"
server_key: "f758378d47f588333ad7c35839673562"
server_key_iv: "219c09497f24cf39065a3b33a44b3597"
server_key_version: "encryptionkey@0"
I20260929 17:55:33.455231 13872 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790704375975685-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790704375975685-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790704375975685-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:55:33.470666 13872 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:55:33.473968 13872 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:55:33.474148 13872 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:55:33.481070 13872 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:33313
I20260929 17:55:33.481101 13940 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:33313 every 8 connection(s)
I20260929 17:55:33.482013 13872 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790704375975685-392-0/minicluster-data/master-2/data/info.pb
I20260929 17:55:33.483687 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskup3Soe/build/debug/bin/kudu as pid 13872
I20260929 17:55:33.483793 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790704375975685-392-0/minicluster-data/master-2/wal/instance
I20260929 17:55:33.484048 392 external_mini_cluster.cc:1428] Setting key dd721da76ddfa21910fde972134d1f48
I20260929 17:55:33.484923 13941 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:55:33.488962 13941 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39429 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 17:55:33 dist-test-slave-53nh krb5kdc[13710](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704532, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 29 17:55:33 dist-test-slave-53nh krb5kdc[13710](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704533, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 17:55:33.498126 13944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:55:33.485340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41818 (local address 127.0.98.60:33313)
0929 17:55:33.485739 (+ 399us) server_negotiation.cc:206] Beginning negotiation
0929 17:55:33.485754 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 17:55:33.485790 (+ 36us) server_negotiation.cc:407] Connection header received
0929 17:55:33.486496 (+ 706us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:55:33.486513 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:55:33.486840 (+ 327us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:55:33.487128 (+ 288us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:55:33.487974 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:55:33.488937 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:55:33.490061 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:55:33.490520 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:55:33.490664 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:55:33.493196 (+ 2532us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:55:33.493202 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:55:33.493212 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:55:33.493238 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:55:33.494993 (+ 1755us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:55:33.495379 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:55:33.495385 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:55:33.495391 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:55:33.495511 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:55:33.495781 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:55:33.495784 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:55:33.495786 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:55:33.496297 (+ 511us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:55:33.496464 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0929 17:55:33.496747 (+ 283us) server_negotiation.cc:299] Negotiation successful
0929 17:55:33.497049 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":265,"thread_start_us":124,"threads_started":1}
I20260929 17:55:33.500582 13794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:55:33.490811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50624 (local address 127.0.98.62:39429)
0929 17:55:33.490963 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0929 17:55:33.490967 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:55:33.491274 (+ 307us) server_negotiation.cc:407] Connection header received
0929 17:55:33.491454 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:55:33.491457 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:55:33.491507 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:55:33.491549 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:55:33.492261 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:55:33.493039 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:55:33.494107 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:55:33.494438 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:55:33.494546 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:55:33.496628 (+ 2082us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:55:33.496632 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:55:33.496635 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:55:33.496661 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:55:33.498080 (+ 1419us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:55:33.499263 (+ 1183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:55:33.499266 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:55:33.499267 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:55:33.499344 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:55:33.499586 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:55:33.499589 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:55:33.499591 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:55:33.499830 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:55:33.499918 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0929 17:55:33.500313 (+ 395us) server_negotiation.cc:299] Negotiation successful
0929 17:55:33.500431 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 17:55:33.501073 13945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:55:33.490768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39429 (local address 127.0.0.1:50624)
0929 17:55:33.491156 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0929 17:55:33.491204 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0929 17:55:33.491353 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:55:33.491689 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:55:33.491693 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:55:33.492023 (+ 330us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:55:33.492144 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:55:33.492149 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:55:33.493203 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:55:33.493207 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:55:33.493989 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:55:33.494001 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:55:33.494558 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:55:33.494561 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:55:33.494645 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:55:33.495064 (+ 419us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:55:33.495076 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:55:33.496488 (+ 1412us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:55:33.498181 (+ 1693us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:55:33.498189 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:55:33.498202 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:55:33.499166 (+ 964us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:55:33.499425 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:55:33.499429 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:55:33.499430 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:55:33.499508 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:55:33.499971 (+ 463us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:55:33.499977 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:55:33.500094 (+ 117us) client_negotiation.cc:769] Sending connection context
0929 17:55:33.500760 (+ 666us) client_negotiation.cc:240] Negotiation successful
0929 17:55:33.500913 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":135,"threads_started":1}
I20260929 17:55:33.502050 13941 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36315 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 17:55:33 dist-test-slave-53nh krb5kdc[13710](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704532, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Sep 29 17:55:33 dist-test-slave-53nh krb5kdc[13710](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704533, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260929 17:55:33.508714 13946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:55:33.500305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50626 (local address 127.0.98.62:39429)
0929 17:55:33.500579 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0929 17:55:33.500585 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 17:55:33.500595 (+ 10us) server_negotiation.cc:407] Connection header received
0929 17:55:33.500622 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:55:33.500624 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:55:33.500658 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:55:33.500735 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:55:33.501463 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:55:33.502068 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:55:33.503030 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:55:33.503255 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:55:33.503340 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:55:33.505669 (+ 2329us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:55:33.505672 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:55:33.505675 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:55:33.505697 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:55:33.507200 (+ 1503us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:55:33.507584 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:55:33.507587 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:55:33.507590 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:55:33.507666 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:55:33.507898 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:55:33.507901 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:55:33.507903 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:55:33.508082 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:55:33.508275 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0929 17:55:33.508407 (+ 132us) server_negotiation.cc:299] Negotiation successful
0929 17:55:33.508525 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":215,"thread_start_us":100,"threads_started":1}
I20260929 17:55:33.509413 13945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:55:33.502527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36315 (local address 127.0.0.1:38326)
0929 17:55:33.502667 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0929 17:55:33.502679 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 17:55:33.502740 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:55:33.502942 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:55:33.502945 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:55:33.503104 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:55:33.503196 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:55:33.503202 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:55:33.503893 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:55:33.503896 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:55:33.504392 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:55:33.504399 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:55:33.504905 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:55:33.504908 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:55:33.504995 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:55:33.505470 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:55:33.505480 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:55:33.506503 (+ 1023us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:55:33.508034 (+ 1531us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:55:33.508038 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:55:33.508040 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:55:33.508219 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:55:33.508461 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:55:33.508464 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:55:33.508466 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:55:33.508538 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:55:33.508894 (+ 356us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:55:33.508897 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:55:33.509036 (+ 139us) client_negotiation.cc:769] Sending connection context
0929 17:55:33.509130 (+ 94us) client_negotiation.cc:240] Negotiation successful
0929 17:55:33.509260 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":86}
I20260929 17:55:33.509428 13871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:55:33.502625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38326 (local address 127.0.98.61:36315)
0929 17:55:33.502726 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0929 17:55:33.502729 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 17:55:33.502741 (+ 12us) server_negotiation.cc:407] Connection header received
0929 17:55:33.502816 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:55:33.502819 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:55:33.502855 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:55:33.502914 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:55:33.503310 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:55:33.503794 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:55:33.504495 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:55:33.504793 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:55:33.504900 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:55:33.506633 (+ 1733us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:55:33.506637 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:55:33.506640 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:55:33.506665 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:55:33.507926 (+ 1261us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:55:33.508301 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:55:33.508304 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:55:33.508306 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:55:33.508376 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:55:33.508612 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:55:33.508615 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:55:33.508616 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:55:33.508775 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:55:33.508852 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0929 17:55:33.509196 (+ 344us) server_negotiation.cc:299] Negotiation successful
0929 17:55:33.509297 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260929 17:55:33.510113 13941 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33313 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 17:55:33 dist-test-slave-53nh krb5kdc[13710](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704533, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 29 17:55:33 dist-test-slave-53nh krb5kdc[13710](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704532, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260929 17:55:33.517918 13945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:55:33.510529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33313 (local address 127.0.0.1:41820)
0929 17:55:33.510628 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0929 17:55:33.510640 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 17:55:33.510705 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:55:33.510919 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:55:33.510921 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:55:33.511038 (+ 117us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:55:33.511103 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:55:33.511107 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:55:33.511863 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:55:33.511867 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:55:33.512514 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:55:33.512523 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:55:33.512951 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:55:33.512955 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:55:33.513059 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:55:33.513399 (+ 340us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:55:33.513412 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:55:33.514531 (+ 1119us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:55:33.516517 (+ 1986us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:55:33.516519 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:55:33.516520 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:55:33.516690 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:55:33.516960 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:55:33.516963 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:55:33.516966 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:55:33.517073 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:55:33.517426 (+ 353us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:55:33.517429 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:55:33.517513 (+ 84us) client_negotiation.cc:769] Sending connection context
0929 17:55:33.517612 (+ 99us) client_negotiation.cc:240] Negotiation successful
0929 17:55:33.517738 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260929 17:55:33.517946 13944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:55:33.510604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41820 (local address 127.0.98.60:33313)
0929 17:55:33.510693 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0929 17:55:33.510696 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 17:55:33.510707 (+ 11us) server_negotiation.cc:407] Connection header received
0929 17:55:33.510785 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:55:33.510787 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:55:33.510826 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:55:33.510894 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:55:33.511185 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:55:33.511777 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:55:33.512616 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:55:33.512851 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:55:33.512926 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:55:33.514664 (+ 1738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:55:33.514667 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:55:33.514669 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:55:33.514691 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:55:33.516401 (+ 1710us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:55:33.516777 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:55:33.516781 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:55:33.516783 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:55:33.516870 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:55:33.517156 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:55:33.517159 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:55:33.517161 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:55:33.517338 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:55:33.517428 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0929 17:55:33.517679 (+ 251us) server_negotiation.cc:299] Negotiation successful
0929 17:55:33.517818 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":38,"server-negotiator.queue_time_us":34}
I20260929 17:55:33.518255 13871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:55:33.509963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38328 (local address 127.0.98.61:36315)
0929 17:55:33.510065 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0929 17:55:33.510069 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:55:33.510080 (+ 11us) server_negotiation.cc:407] Connection header received
0929 17:55:33.510105 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:55:33.510107 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:55:33.510137 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:55:33.510195 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:55:33.510794 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:55:33.511427 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:55:33.512278 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:55:33.512484 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:55:33.512555 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:55:33.515220 (+ 2665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:55:33.515241 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:55:33.515244 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:55:33.515272 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:55:33.516736 (+ 1464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:55:33.517100 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:55:33.517103 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:55:33.517105 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:55:33.517179 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:55:33.517398 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:55:33.517401 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:55:33.517403 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:55:33.517561 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:55:33.517775 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0929 17:55:33.517871 (+ 96us) server_negotiation.cc:299] Negotiation successful
0929 17:55:33.517981 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 17:55:33.521683 13941 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e760625c2e54fe4bc2d405a34eb439a: Bootstrap starting.
I20260929 17:55:33.521900 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 17:55:33.524048 13941 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3e760625c2e54fe4bc2d405a34eb439a: Neither blocks nor log segments found. Creating new log.
I20260929 17:55:33.524860 13941 log.cc:824] T 00000000000000000000000000000000 P 3e760625c2e54fe4bc2d405a34eb439a: Log is configured to *not* fsync() on all Append() calls
I20260929 17:55:33.525837 13866 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33313 } attempt: 3
I20260929 17:55:33.526872 13941 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e760625c2e54fe4bc2d405a34eb439a: No bootstrap required, opened a new log
I20260929 17:55:33.530975 13941 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3e760625c2e54fe4bc2d405a34eb439a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5142b27fb864cd993938fa340d3bc9a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39429 } } peers { permanent_uuid: "f39aedce090a401cb3a32682669b1876" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36315 } } peers { permanent_uuid: "3e760625c2e54fe4bc2d405a34eb439a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33313 } }
I20260929 17:55:33.531323 13941 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3e760625c2e54fe4bc2d405a34eb439a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:55:33.531420 13941 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3e760625c2e54fe4bc2d405a34eb439a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e760625c2e54fe4bc2d405a34eb439a, State: Initialized, Role: FOLLOWER
I20260929 17:55:33.531997 13941 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3e760625c2e54fe4bc2d405a34eb439a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5142b27fb864cd993938fa340d3bc9a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39429 } } peers { permanent_uuid: "f39aedce090a401cb3a32682669b1876" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36315 } } peers { permanent_uuid: "3e760625c2e54fe4bc2d405a34eb439a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33313 } }
I20260929 17:55:33.533174 13946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:55:33.523839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50632 (local address 127.0.98.62:39429)
0929 17:55:33.523956 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0929 17:55:33.523960 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:55:33.524164 (+ 204us) server_negotiation.cc:407] Connection header received
0929 17:55:33.524365 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:55:33.524369 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:55:33.524426 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:55:33.524483 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:55:33.525446 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:55:33.526174 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:55:33.527220 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:55:33.527542 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:55:33.527630 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:55:33.529287 (+ 1657us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:55:33.529290 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:55:33.529293 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:55:33.529321 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:55:33.531317 (+ 1996us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:55:33.531757 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:55:33.531761 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:55:33.531763 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:55:33.531863 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:55:33.532158 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:55:33.532161 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:55:33.532163 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:55:33.532365 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:55:33.532463 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0929 17:55:33.532826 (+ 363us) server_negotiation.cc:299] Negotiation successful
0929 17:55:33.532956 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 17:55:33.533692 13941 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3e760625c2e54fe4bc2d405a34eb439a [sys.catalog]: configured and running, proceeding with master startup.
I20260929 17:55:33.533962 13791 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33313 } attempt: 3
I20260929 17:55:33.534013 13944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:55:33.524165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41824 (local address 127.0.98.60:33313)
0929 17:55:33.524272 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0929 17:55:33.524276 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:55:33.524788 (+ 512us) server_negotiation.cc:407] Connection header received
0929 17:55:33.524894 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:55:33.524897 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:55:33.524949 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:55:33.525094 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:55:33.525781 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:55:33.526454 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:55:33.527468 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:55:33.527751 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:55:33.527822 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:55:33.529928 (+ 2106us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:55:33.529932 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:55:33.529934 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:55:33.529968 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:55:33.531765 (+ 1797us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:55:33.532212 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:55:33.532215 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:55:33.532217 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:55:33.532293 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:55:33.533054 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:55:33.533057 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:55:33.533059 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:55:33.533247 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:55:33.533340 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0929 17:55:33.533773 (+ 433us) server_negotiation.cc:299] Negotiation successful
0929 17:55:33.533873 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260929 17:55:33.535313 13871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:55:33.524427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38332 (local address 127.0.98.61:36315)
0929 17:55:33.524523 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0929 17:55:33.524526 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 17:55:33.527389 (+ 2863us) server_negotiation.cc:407] Connection header received
0929 17:55:33.527542 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:55:33.527545 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:55:33.527590 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:55:33.527635 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:55:33.528467 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:55:33.529207 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:55:33.530223 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:55:33.530516 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:55:33.530597 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:55:33.532092 (+ 1495us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:55:33.532096 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:55:33.532098 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:55:33.532123 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:55:33.533581 (+ 1458us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:55:33.534016 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:55:33.534020 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:55:33.534022 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:55:33.534099 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:55:33.534394 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:55:33.534397 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:55:33.534399 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:55:33.534591 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:55:33.534699 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0929 17:55:33.535053 (+ 354us) server_negotiation.cc:299] Negotiation successful
0929 17:55:33.535171 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
W20260929 17:55:33.539425 13967 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3e760625c2e54fe4bc2d405a34eb439a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 17:55:33.539521 13967 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 17:55:33.539691 13956 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e760625c2e54fe4bc2d405a34eb439a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5142b27fb864cd993938fa340d3bc9a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39429 } } peers { permanent_uuid: "f39aedce090a401cb3a32682669b1876" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36315 } } peers { permanent_uuid: "3e760625c2e54fe4bc2d405a34eb439a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33313 } } }
I20260929 17:55:33.539837 13956 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e760625c2e54fe4bc2d405a34eb439a [sys.catalog]: This master's current role is: FOLLOWER
Sep 29 17:55:33 dist-test-slave-53nh krb5kdc[13710](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704533, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 29 17:55:33 dist-test-slave-53nh krb5kdc[13710](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704533, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260929 17:55:33.537247 13872 master_runner.cc:186] Error getting master registration for 127.0.98.62:39429: OK, instance_id { permanent_uuid: "c5142b27fb864cd993938fa340d3bc9a" instance_seqno: 1790704533073202 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 17:55:33.542030 13872 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 17:55:33.544549 13795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:55:33.534830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33313 (local address 127.0.0.1:41838)
0929 17:55:33.534984 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0929 17:55:33.534997 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 17:55:33.535054 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:55:33.535775 (+ 721us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:55:33.535779 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:55:33.535938 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:55:33.536032 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:55:33.536038 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:55:33.537185 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:55:33.537188 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:55:33.538024 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:55:33.538034 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:55:33.538791 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:55:33.538795 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:55:33.538887 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:55:33.539984 (+ 1097us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:55:33.539998 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:55:33.541414 (+ 1416us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:55:33.543146 (+ 1732us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:55:33.543148 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:55:33.543151 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:55:33.543329 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:55:33.543587 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:55:33.543589 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:55:33.543591 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:55:33.543667 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:55:33.544079 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:55:33.544082 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:55:33.544172 (+ 90us) client_negotiation.cc:769] Sending connection context
0929 17:55:33.544253 (+ 81us) client_negotiation.cc:240] Negotiation successful
0929 17:55:33.544389 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91}
I20260929 17:55:33.544585 13944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:55:33.535399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41838 (local address 127.0.98.60:33313)
0929 17:55:33.535493 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0929 17:55:33.535497 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:55:33.535509 (+ 12us) server_negotiation.cc:407] Connection header received
0929 17:55:33.535537 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:55:33.535540 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:55:33.535573 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:55:33.535646 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:55:33.536125 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:55:33.536797 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:55:33.538154 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:55:33.538589 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:55:33.538681 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:55:33.541569 (+ 2888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:55:33.541573 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:55:33.541575 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:55:33.541600 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:55:33.543038 (+ 1438us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:55:33.543416 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:55:33.543419 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:55:33.543421 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:55:33.543508 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:55:33.543765 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:55:33.543769 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:55:33.543771 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:55:33.543989 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:55:33.544073 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0929 17:55:33.544316 (+ 243us) server_negotiation.cc:299] Negotiation successful
0929 17:55:33.544459 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":58,"server-negotiator.queue_time_us":39}
I20260929 17:55:33.546415 13869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:55:33.526196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33313 (local address 127.0.0.1:41836)
0929 17:55:33.526463 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0929 17:55:33.526477 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 17:55:33.526546 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:55:33.530900 (+ 4354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:55:33.530904 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:55:33.531064 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:55:33.531157 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:55:33.531163 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:55:33.537258 (+ 6095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:55:33.537262 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:55:33.538050 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:55:33.538059 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:55:33.538748 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:55:33.538752 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:55:33.538835 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:55:33.539417 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:55:33.539430 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:55:33.540862 (+ 1432us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:55:33.544748 (+ 3886us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:55:33.544752 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:55:33.544754 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:55:33.545037 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:55:33.545360 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:55:33.545363 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:55:33.545366 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:55:33.545453 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:55:33.545935 (+ 482us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:55:33.545938 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:55:33.546037 (+ 99us) client_negotiation.cc:769] Sending connection context
0929 17:55:33.546125 (+ 88us) client_negotiation.cc:240] Negotiation successful
0929 17:55:33.546257 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":198}
I20260929 17:55:33.547034 13955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:55:33.526665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41836 (local address 127.0.98.60:33313)
0929 17:55:33.530400 (+ 3735us) server_negotiation.cc:206] Beginning negotiation
0929 17:55:33.530407 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 17:55:33.530418 (+ 11us) server_negotiation.cc:407] Connection header received
0929 17:55:33.530449 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:55:33.530451 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:55:33.530488 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:55:33.532400 (+ 1912us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:55:33.532838 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:55:33.537140 (+ 4302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:55:33.538171 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:55:33.538613 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:55:33.539098 (+ 485us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:55:33.541035 (+ 1937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:55:33.541039 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:55:33.541042 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:55:33.541067 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:55:33.544630 (+ 3563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:55:33.545152 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:55:33.545155 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:55:33.545157 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:55:33.545254 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:55:33.545558 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:55:33.545561 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:55:33.545563 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:55:33.545808 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:55:33.546657 (+ 849us) server_negotiation.cc:1035] Waiting for connection context
0929 17:55:33.546743 (+ 86us) server_negotiation.cc:299] Negotiation successful
0929 17:55:33.546868 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":31,"server-negotiator.queue_time_us":3653,"thread_start_us":82,"threads_started":1}
I20260929 17:55:33.547392 13791 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c5142b27fb864cd993938fa340d3bc9a: Bootstrap starting.
I20260929 17:55:33.549599 13791 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c5142b27fb864cd993938fa340d3bc9a: Neither blocks nor log segments found. Creating new log.
I20260929 17:55:33.550031 13866 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f39aedce090a401cb3a32682669b1876: Bootstrap starting.
I20260929 17:55:33.550288 13791 log.cc:824] T 00000000000000000000000000000000 P c5142b27fb864cd993938fa340d3bc9a: Log is configured to *not* fsync() on all Append() calls
I20260929 17:55:33.551961 13791 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c5142b27fb864cd993938fa340d3bc9a: No bootstrap required, opened a new log
I20260929 17:55:33.552554 13866 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f39aedce090a401cb3a32682669b1876: Neither blocks nor log segments found. Creating new log.
I20260929 17:55:33.553490 13866 log.cc:824] T 00000000000000000000000000000000 P f39aedce090a401cb3a32682669b1876: Log is configured to *not* fsync() on all Append() calls
I20260929 17:55:33.555595 13866 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f39aedce090a401cb3a32682669b1876: No bootstrap required, opened a new log
I20260929 17:55:33.555596 13791 raft_consensus.cc:348] T 00000000000000000000000000000000 P c5142b27fb864cd993938fa340d3bc9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5142b27fb864cd993938fa340d3bc9a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39429 } } peers { permanent_uuid: "f39aedce090a401cb3a32682669b1876" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36315 } } peers { permanent_uuid: "3e760625c2e54fe4bc2d405a34eb439a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33313 } }
I20260929 17:55:33.555758 13791 raft_consensus.cc:374] T 00000000000000000000000000000000 P c5142b27fb864cd993938fa340d3bc9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:55:33.555794 13791 raft_consensus.cc:729] T 00000000000000000000000000000000 P c5142b27fb864cd993938fa340d3bc9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5142b27fb864cd993938fa340d3bc9a, State: Initialized, Role: FOLLOWER
I20260929 17:55:33.556167 13791 consensus_queue.cc:260] T 00000000000000000000000000000000 P c5142b27fb864cd993938fa340d3bc9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5142b27fb864cd993938fa340d3bc9a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39429 } } peers { permanent_uuid: "f39aedce090a401cb3a32682669b1876" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36315 } } peers { permanent_uuid: "3e760625c2e54fe4bc2d405a34eb439a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33313 } }
I20260929 17:55:33.556505 13971 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5142b27fb864cd993938fa340d3bc9a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5142b27fb864cd993938fa340d3bc9a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39429 } } peers { permanent_uuid: "f39aedce090a401cb3a32682669b1876" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36315 } } peers { permanent_uuid: "3e760625c2e54fe4bc2d405a34eb439a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33313 } } }
I20260929 17:55:33.556615 13971 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5142b27fb864cd993938fa340d3bc9a [sys.catalog]: This master's current role is: FOLLOWER
I20260929 17:55:33.557240 13791 sys_catalog.cc:564] T 00000000000000000000000000000000 P c5142b27fb864cd993938fa340d3bc9a [sys.catalog]: configured and running, proceeding with master startup.
I20260929 17:55:33.560146 13866 raft_consensus.cc:348] T 00000000000000000000000000000000 P f39aedce090a401cb3a32682669b1876 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5142b27fb864cd993938fa340d3bc9a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39429 } } peers { permanent_uuid: "f39aedce090a401cb3a32682669b1876" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36315 } } peers { permanent_uuid: "3e760625c2e54fe4bc2d405a34eb439a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33313 } }
I20260929 17:55:33.560400 13866 raft_consensus.cc:374] T 00000000000000000000000000000000 P f39aedce090a401cb3a32682669b1876 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:55:33.560498 13866 raft_consensus.cc:729] T 00000000000000000000000000000000 P f39aedce090a401cb3a32682669b1876 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f39aedce090a401cb3a32682669b1876, State: Initialized, Role: FOLLOWER
I20260929 17:55:33.560992 13866 consensus_queue.cc:260] T 00000000000000000000000000000000 P f39aedce090a401cb3a32682669b1876 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5142b27fb864cd993938fa340d3bc9a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39429 } } peers { permanent_uuid: "f39aedce090a401cb3a32682669b1876" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36315 } } peers { permanent_uuid: "3e760625c2e54fe4bc2d405a34eb439a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33313 } }
I20260929 17:55:33.561589 13983 sys_catalog.cc:455] T 00000000000000000000000000000000 P f39aedce090a401cb3a32682669b1876 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5142b27fb864cd993938fa340d3bc9a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39429 } } peers { permanent_uuid: "f39aedce090a401cb3a32682669b1876" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36315 } } peers { permanent_uuid: "3e760625c2e54fe4bc2d405a34eb439a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33313 } } }
I20260929 17:55:33.561774 13983 sys_catalog.cc:458] T 00000000000000000000000000000000 P f39aedce090a401cb3a32682669b1876 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 17:55:33.562263 13866 sys_catalog.cc:564] T 00000000000000000000000000000000 P f39aedce090a401cb3a32682669b1876 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 17:55:33.561417 13723 master_runner.cc:186] Error getting master registration for 127.0.98.61:36315: OK, instance_id { permanent_uuid: "f39aedce090a401cb3a32682669b1876" instance_seqno: 1790704533275468 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 17:55:33.564916 13723 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260929 17:55:33.568485 13982 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c5142b27fb864cd993938fa340d3bc9a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 17:55:33.568719 13982 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260929 17:55:33.569871 13744 tablet.cc:2369] T 00000000000000000000000000000000 P c5142b27fb864cd993938fa340d3bc9a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 17:55:33.570341 13798 master_runner.cc:263] No leader master found from master f39aedce090a401cb3a32682669b1876
I20260929 17:55:33.570405 13798 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260929 17:55:33.572547 13994 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f39aedce090a401cb3a32682669b1876: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 17:55:33.572634 13994 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260929 17:55:33.725515 13894 tablet.cc:2369] T 00000000000000000000000000000000 P 3e760625c2e54fe4bc2d405a34eb439a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 17:55:33.772215 13819 tablet.cc:2369] T 00000000000000000000000000000000 P f39aedce090a401cb3a32682669b1876: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 17:55:33.883812 13983 raft_consensus.cc:482] T 00000000000000000000000000000000 P f39aedce090a401cb3a32682669b1876 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 17:55:33.884037 13983 raft_consensus.cc:504] T 00000000000000000000000000000000 P f39aedce090a401cb3a32682669b1876 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5142b27fb864cd993938fa340d3bc9a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39429 } } peers { permanent_uuid: "f39aedce090a401cb3a32682669b1876" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36315 } } peers { permanent_uuid: "3e760625c2e54fe4bc2d405a34eb439a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33313 } }
I20260929 17:55:33.885027 13983 leader_election.cc:288] T 00000000000000000000000000000000 P f39aedce090a401cb3a32682669b1876 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c5142b27fb864cd993938fa340d3bc9a (127.0.98.62:39429), 3e760625c2e54fe4bc2d405a34eb439a (127.0.98.60:33313)
I20260929 17:55:33.885594 13765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f39aedce090a401cb3a32682669b1876" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5142b27fb864cd993938fa340d3bc9a" is_pre_election: true
I20260929 17:55:33.885610 13915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f39aedce090a401cb3a32682669b1876" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e760625c2e54fe4bc2d405a34eb439a" is_pre_election: true
I20260929 17:55:33.885886 13765 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c5142b27fb864cd993938fa340d3bc9a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f39aedce090a401cb3a32682669b1876 in term 0.
I20260929 17:55:33.885886 13915 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3e760625c2e54fe4bc2d405a34eb439a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f39aedce090a401cb3a32682669b1876 in term 0.
I20260929 17:55:33.886291 13814 leader_election.cc:302] T 00000000000000000000000000000000 P f39aedce090a401cb3a32682669b1876 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e760625c2e54fe4bc2d405a34eb439a, f39aedce090a401cb3a32682669b1876; no voters:
I20260929 17:55:33.886562 13983 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f39aedce090a401cb3a32682669b1876 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 17:55:33.886664 13983 raft_consensus.cc:482] T 00000000000000000000000000000000 P f39aedce090a401cb3a32682669b1876 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 17:55:33.886713 13983 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f39aedce090a401cb3a32682669b1876 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:55:33.887727 13983 raft_consensus.cc:504] T 00000000000000000000000000000000 P f39aedce090a401cb3a32682669b1876 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5142b27fb864cd993938fa340d3bc9a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39429 } } peers { permanent_uuid: "f39aedce090a401cb3a32682669b1876" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36315 } } peers { permanent_uuid: "3e760625c2e54fe4bc2d405a34eb439a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33313 } }
I20260929 17:55:33.888051 13983 leader_election.cc:288] T 00000000000000000000000000000000 P f39aedce090a401cb3a32682669b1876 [CANDIDATE]: Term 1 election: Requested vote from peers c5142b27fb864cd993938fa340d3bc9a (127.0.98.62:39429), 3e760625c2e54fe4bc2d405a34eb439a (127.0.98.60:33313)
I20260929 17:55:33.888360 13765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f39aedce090a401cb3a32682669b1876" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5142b27fb864cd993938fa340d3bc9a"
I20260929 17:55:33.888368 13915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f39aedce090a401cb3a32682669b1876" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e760625c2e54fe4bc2d405a34eb439a"
I20260929 17:55:33.888481 13915 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3e760625c2e54fe4bc2d405a34eb439a [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:55:33.888489 13765 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c5142b27fb864cd993938fa340d3bc9a [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:55:33.889614 13765 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c5142b27fb864cd993938fa340d3bc9a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f39aedce090a401cb3a32682669b1876 in term 1.
I20260929 17:55:33.889708 13915 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3e760625c2e54fe4bc2d405a34eb439a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f39aedce090a401cb3a32682669b1876 in term 1.
I20260929 17:55:33.889919 13815 leader_election.cc:302] T 00000000000000000000000000000000 P f39aedce090a401cb3a32682669b1876 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5142b27fb864cd993938fa340d3bc9a, f39aedce090a401cb3a32682669b1876; no voters:
I20260929 17:55:33.890379 13983 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f39aedce090a401cb3a32682669b1876 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 17:55:33.890604 13983 raft_consensus.cc:686] T 00000000000000000000000000000000 P f39aedce090a401cb3a32682669b1876 [term 1 LEADER]: Becoming Leader. State: Replica: f39aedce090a401cb3a32682669b1876, State: Running, Role: LEADER
I20260929 17:55:33.890903 13983 consensus_queue.cc:237] T 00000000000000000000000000000000 P f39aedce090a401cb3a32682669b1876 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5142b27fb864cd993938fa340d3bc9a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39429 } } peers { permanent_uuid: "f39aedce090a401cb3a32682669b1876" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36315 } } peers { permanent_uuid: "3e760625c2e54fe4bc2d405a34eb439a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33313 } }
I20260929 17:55:33.892647 13997 sys_catalog.cc:455] T 00000000000000000000000000000000 P f39aedce090a401cb3a32682669b1876 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f39aedce090a401cb3a32682669b1876. Latest consensus state: current_term: 1 leader_uuid: "f39aedce090a401cb3a32682669b1876" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5142b27fb864cd993938fa340d3bc9a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39429 } } peers { permanent_uuid: "f39aedce090a401cb3a32682669b1876" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36315 } } peers { permanent_uuid: "3e760625c2e54fe4bc2d405a34eb439a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33313 } } }
I20260929 17:55:33.892786 13997 sys_catalog.cc:458] T 00000000000000000000000000000000 P f39aedce090a401cb3a32682669b1876 [sys.catalog]: This master's current role is: LEADER
I20260929 17:55:33.893153 13999 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 17:55:33.893965 13999 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 17:55:33.898404 13765 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c5142b27fb864cd993938fa340d3bc9a [term 1 FOLLOWER]: Refusing update from remote peer f39aedce090a401cb3a32682669b1876: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 17:55:33.898406 13915 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3e760625c2e54fe4bc2d405a34eb439a [term 1 FOLLOWER]: Refusing update from remote peer f39aedce090a401cb3a32682669b1876: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 17:55:33.899021 13997 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f39aedce090a401cb3a32682669b1876 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5142b27fb864cd993938fa340d3bc9a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39429 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 17:55:33.899170 13983 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f39aedce090a401cb3a32682669b1876 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e760625c2e54fe4bc2d405a34eb439a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 17:55:33.903271 13956 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e760625c2e54fe4bc2d405a34eb439a [sys.catalog]: SysCatalogTable state changed. Reason: New leader f39aedce090a401cb3a32682669b1876. Latest consensus state: current_term: 1 leader_uuid: "f39aedce090a401cb3a32682669b1876" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5142b27fb864cd993938fa340d3bc9a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39429 } } peers { permanent_uuid: "f39aedce090a401cb3a32682669b1876" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36315 } } peers { permanent_uuid: "3e760625c2e54fe4bc2d405a34eb439a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33313 } } }
I20260929 17:55:33.903271 13971 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5142b27fb864cd993938fa340d3bc9a [sys.catalog]: SysCatalogTable state changed. Reason: New leader f39aedce090a401cb3a32682669b1876. Latest consensus state: current_term: 1 leader_uuid: "f39aedce090a401cb3a32682669b1876" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5142b27fb864cd993938fa340d3bc9a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39429 } } peers { permanent_uuid: "f39aedce090a401cb3a32682669b1876" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36315 } } peers { permanent_uuid: "3e760625c2e54fe4bc2d405a34eb439a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33313 } } }
I20260929 17:55:33.903394 13956 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e760625c2e54fe4bc2d405a34eb439a [sys.catalog]: This master's current role is: FOLLOWER
I20260929 17:55:33.903394 13971 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5142b27fb864cd993938fa340d3bc9a [sys.catalog]: This master's current role is: FOLLOWER
I20260929 17:55:33.905457 13983 sys_catalog.cc:455] T 00000000000000000000000000000000 P f39aedce090a401cb3a32682669b1876 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f39aedce090a401cb3a32682669b1876" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5142b27fb864cd993938fa340d3bc9a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39429 } } peers { permanent_uuid: "f39aedce090a401cb3a32682669b1876" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36315 } } peers { permanent_uuid: "3e760625c2e54fe4bc2d405a34eb439a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33313 } } }
I20260929 17:55:33.905573 13983 sys_catalog.cc:458] T 00000000000000000000000000000000 P f39aedce090a401cb3a32682669b1876 [sys.catalog]: This master's current role is: LEADER
I20260929 17:55:33.905507 14005 sys_catalog.cc:455] T 00000000000000000000000000000000 P f39aedce090a401cb3a32682669b1876 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f39aedce090a401cb3a32682669b1876" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5142b27fb864cd993938fa340d3bc9a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39429 } } peers { permanent_uuid: "f39aedce090a401cb3a32682669b1876" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36315 } } peers { permanent_uuid: "3e760625c2e54fe4bc2d405a34eb439a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33313 } } }
I20260929 17:55:33.905615 14005 sys_catalog.cc:458] T 00000000000000000000000000000000 P f39aedce090a401cb3a32682669b1876 [sys.catalog]: This master's current role is: LEADER
I20260929 17:55:33.906167 13956 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e760625c2e54fe4bc2d405a34eb439a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f39aedce090a401cb3a32682669b1876" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5142b27fb864cd993938fa340d3bc9a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39429 } } peers { permanent_uuid: "f39aedce090a401cb3a32682669b1876" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36315 } } peers { permanent_uuid: "3e760625c2e54fe4bc2d405a34eb439a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33313 } } }
I20260929 17:55:33.906281 13956 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e760625c2e54fe4bc2d405a34eb439a [sys.catalog]: This master's current role is: FOLLOWER
I20260929 17:55:33.906477 14001 mvcc.cc:205] Tried to move back new op lower bound from 7334725770841370624 to 7334725770819485696. Current Snapshot: MvccSnapshot[applied={T|T < 7334725770841370624}]
I20260929 17:55:33.906651 13999 catalog_manager.cc:1347] Generated new cluster ID: eae44a7b9d6a4b7b9312d0c4972f290a
I20260929 17:55:33.906700 13999 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 17:55:33.907092 13971 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5142b27fb864cd993938fa340d3bc9a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f39aedce090a401cb3a32682669b1876" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5142b27fb864cd993938fa340d3bc9a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39429 } } peers { permanent_uuid: "f39aedce090a401cb3a32682669b1876" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36315 } } peers { permanent_uuid: "3e760625c2e54fe4bc2d405a34eb439a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33313 } } }
I20260929 17:55:33.907203 13971 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5142b27fb864cd993938fa340d3bc9a [sys.catalog]: This master's current role is: FOLLOWER
I20260929 17:55:33.916896 13999 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 17:55:33.918080 13999 catalog_manager.cc:1504] Loading token signing keys...
I20260929 17:55:33.926782 13999 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f39aedce090a401cb3a32682669b1876: Generated new TSK 0
I20260929 17:55:33.927771 13999 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 17:55:34.540874 13967 catalog_manager.cc:1259] Loaded cluster ID: eae44a7b9d6a4b7b9312d0c4972f290a
I20260929 17:55:34.540956 13967 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 3e760625c2e54fe4bc2d405a34eb439a: loading cluster ID for follower catalog manager: success
I20260929 17:55:34.542672 13967 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 3e760625c2e54fe4bc2d405a34eb439a: acquiring CA information for follower catalog manager: success
I20260929 17:55:34.543885 13967 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 3e760625c2e54fe4bc2d405a34eb439a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 17:55:34.570103 13982 catalog_manager.cc:1259] Loaded cluster ID: eae44a7b9d6a4b7b9312d0c4972f290a
I20260929 17:55:34.570187 13982 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c5142b27fb864cd993938fa340d3bc9a: loading cluster ID for follower catalog manager: success
I20260929 17:55:34.571846 13982 catalog_manager.cc:1574] T 00000000000000000000000000000000 P c5142b27fb864cd993938fa340d3bc9a: acquiring CA information for follower catalog manager: success
I20260929 17:55:34.573128 13982 catalog_manager.cc:1602] T 00000000000000000000000000000000 P c5142b27fb864cd993938fa340d3bc9a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 17:55:34.999274 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskup3Soe/build/debug/bin/kudu with pid 13723
I20260929 17:55:35.004935 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskup3Soe/build/debug/bin/kudu with pid 13798
I20260929 17:55:35.009856 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskup3Soe/build/debug/bin/kudu with pid 13872
2026-09-29T17:55:35Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3177 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790704375975685-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 17:55:35 dist-test-slave-53nh krb5kdc[14015](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 17:55:35 dist-test-slave-53nh krb5kdc[14015](info): set up 2 sockets
Sep 29 17:55:35 dist-test-slave-53nh krb5kdc[14015](info): commencing operation
krb5kdc: starting...
W20260929 17:55:35.949352 392 mini_kdc.cc:121] Time spent starting KDC: real 0.920s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790704375975685-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790704375975685-392-0/krb5kdc/test-user.keytab.
Sep 29 17:55:36 dist-test-slave-53nh krb5kdc[14015](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704536, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T17:55:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T17:55:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790704375975685-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790704375975685-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 17:55:36.068145 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskup3Soe/build/debug/bin/kudu
/tmp/dist-test-taskup3Soe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790704375975685-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790704375975685-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790704375975685-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790704375975685-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33405
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790704375975685-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33405
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790704375975685-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790704375975685-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790704375975685-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:55:36.177101 14028 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:55:36.177367 14028 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:55:36.177407 14028 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:55:36.181344 14028 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 17:55:36.181406 14028 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:55:36.181425 14028 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:55:36.181442 14028 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 17:55:36.181455 14028 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 17:55:36.186620 14028 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790704375975685-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790704375975685-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33405
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790704375975685-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33405
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790704375975685-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14028
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790704375975685-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:55:36.187609 14028 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:55:36.188553 14028 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:55:36.196617 14033 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:55:36.196674 14028 server_base.cc:1034] running on GCE node
W20260929 17:55:36.196612 14034 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:55:36.196589 14036 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:55:36.197351 14028 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:55:36.198352 14028 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:55:36.199512 14028 hybrid_clock.cc:648] HybridClock initialized: now 1790704536199493 us; error 30 us; skew 500 ppm
Sep 29 17:55:36 dist-test-slave-53nh krb5kdc[14015](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704536, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:55:36.202162 14028 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 17:55:36.203001 14028 webserver.cc:460] Webserver started at http://127.0.98.62:40099/ using document root <none> and password file <none>
I20260929 17:55:36.203532 14028 fs_manager.cc:359] Metadata directory not provided
I20260929 17:55:36.203593 14028 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:55:36.203781 14028 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:55:36.205801 14028 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790704375975685-392-0/minicluster-data/master-0/data/instance:
uuid: "5e0cc1a1df4444808156bdc323f6a303"
format_stamp: "Formatted at 2026-09-29 17:55:36 on dist-test-slave-53nh"
server_key: "348ed8f19df0628040f6c64691dddbe6"
server_key_iv: "a6cb8d80a56320ed86e629264d627aae"
server_key_version: "encryptionkey@0"
I20260929 17:55:36.206321 14028 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790704375975685-392-0/minicluster-data/master-0/wal/instance:
uuid: "5e0cc1a1df4444808156bdc323f6a303"
format_stamp: "Formatted at 2026-09-29 17:55:36 on dist-test-slave-53nh"
server_key: "348ed8f19df0628040f6c64691dddbe6"
server_key_iv: "a6cb8d80a56320ed86e629264d627aae"
server_key_version: "encryptionkey@0"
I20260929 17:55:36.209875 14028 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 17:55:36.212250 14043 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:55:36.213486 14028 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 17:55:36.213606 14028 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790704375975685-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790704375975685-392-0/minicluster-data/master-0/wal
uuid: "5e0cc1a1df4444808156bdc323f6a303"
format_stamp: "Formatted at 2026-09-29 17:55:36 on dist-test-slave-53nh"
server_key: "348ed8f19df0628040f6c64691dddbe6"
server_key_iv: "a6cb8d80a56320ed86e629264d627aae"
server_key_version: "encryptionkey@0"
I20260929 17:55:36.213701 14028 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790704375975685-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790704375975685-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790704375975685-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:55:36.230397 14028 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:55:36.233367 14028 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:55:36.233546 14028 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:55:36.240109 14028 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33405
I20260929 17:55:36.240152 14095 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33405 every 8 connection(s)
I20260929 17:55:36.241114 14028 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790704375975685-392-0/minicluster-data/master-0/data/info.pb
I20260929 17:55:36.244016 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskup3Soe/build/debug/bin/kudu as pid 14028
I20260929 17:55:36.244118 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790704375975685-392-0/minicluster-data/master-0/wal/instance
I20260929 17:55:36.244148 14096 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:55:36.244370 392 external_mini_cluster.cc:1428] Setting key 1ea4f2dbb7da48aa6adcec6cbbf7f1cc
I20260929 17:55:36.250734 14096 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e0cc1a1df4444808156bdc323f6a303: Bootstrap starting.
Sep 29 17:55:36 dist-test-slave-53nh krb5kdc[14015](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704536, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 17:55:36.253266 14096 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5e0cc1a1df4444808156bdc323f6a303: Neither blocks nor log segments found. Creating new log.
I20260929 17:55:36.253999 14096 log.cc:824] T 00000000000000000000000000000000 P 5e0cc1a1df4444808156bdc323f6a303: Log is configured to *not* fsync() on all Append() calls
I20260929 17:55:36.255952 14096 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e0cc1a1df4444808156bdc323f6a303: No bootstrap required, opened a new log
I20260929 17:55:36.258265 14099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:55:36.245731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55210 (local address 127.0.98.62:33405)
0929 17:55:36.246239 (+ 508us) server_negotiation.cc:206] Beginning negotiation
0929 17:55:36.246254 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 17:55:36.246292 (+ 38us) server_negotiation.cc:407] Connection header received
0929 17:55:36.247054 (+ 762us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:55:36.247076 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:55:36.247428 (+ 352us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:55:36.247747 (+ 319us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:55:36.248567 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:55:36.249637 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:55:36.250472 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:55:36.250974 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:55:36.251114 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:55:36.253711 (+ 2597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:55:36.253716 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:55:36.253730 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:55:36.253760 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:55:36.255528 (+ 1768us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:55:36.255961 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:55:36.255966 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:55:36.255970 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:55:36.256101 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:55:36.256370 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:55:36.256373 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:55:36.256375 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:55:36.256764 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:55:36.256941 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0929 17:55:36.257239 (+ 298us) server_negotiation.cc:299] Negotiation successful
0929 17:55:36.257533 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":361,"thread_start_us":152,"threads_started":1}
I20260929 17:55:36.259871 14096 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5e0cc1a1df4444808156bdc323f6a303 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e0cc1a1df4444808156bdc323f6a303" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33405 } }
I20260929 17:55:36.260102 14096 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5e0cc1a1df4444808156bdc323f6a303 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:55:36.260160 14096 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5e0cc1a1df4444808156bdc323f6a303 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e0cc1a1df4444808156bdc323f6a303, State: Initialized, Role: FOLLOWER
I20260929 17:55:36.260617 14096 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5e0cc1a1df4444808156bdc323f6a303 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e0cc1a1df4444808156bdc323f6a303" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33405 } }
I20260929 17:55:36.260769 14096 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5e0cc1a1df4444808156bdc323f6a303 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 17:55:36.260834 14096 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5e0cc1a1df4444808156bdc323f6a303 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 17:55:36.260937 14096 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5e0cc1a1df4444808156bdc323f6a303 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:55:36.261415 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 17:55:36.262131 14096 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5e0cc1a1df4444808156bdc323f6a303 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e0cc1a1df4444808156bdc323f6a303" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33405 } }
I20260929 17:55:36.262480 14096 leader_election.cc:302] T 00000000000000000000000000000000 P 5e0cc1a1df4444808156bdc323f6a303 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5e0cc1a1df4444808156bdc323f6a303; no voters:
I20260929 17:55:36.262841 14096 leader_election.cc:288] T 00000000000000000000000000000000 P 5e0cc1a1df4444808156bdc323f6a303 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 17:55:36.262919 14101 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5e0cc1a1df4444808156bdc323f6a303 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 17:55:36.263363 14101 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5e0cc1a1df4444808156bdc323f6a303 [term 1 LEADER]: Becoming Leader. State: Replica: 5e0cc1a1df4444808156bdc323f6a303, State: Running, Role: LEADER
I20260929 17:55:36.263679 14101 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5e0cc1a1df4444808156bdc323f6a303 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e0cc1a1df4444808156bdc323f6a303" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33405 } }
I20260929 17:55:36.264086 14096 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5e0cc1a1df4444808156bdc323f6a303 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 17:55:36.265226 14102 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e0cc1a1df4444808156bdc323f6a303 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5e0cc1a1df4444808156bdc323f6a303" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e0cc1a1df4444808156bdc323f6a303" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33405 } } }
I20260929 17:55:36.265353 14102 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e0cc1a1df4444808156bdc323f6a303 [sys.catalog]: This master's current role is: LEADER
I20260929 17:55:36.265718 14110 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 17:55:36.265695 14103 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e0cc1a1df4444808156bdc323f6a303 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5e0cc1a1df4444808156bdc323f6a303. Latest consensus state: current_term: 1 leader_uuid: "5e0cc1a1df4444808156bdc323f6a303" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e0cc1a1df4444808156bdc323f6a303" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33405 } } }
I20260929 17:55:36.266345 14103 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e0cc1a1df4444808156bdc323f6a303 [sys.catalog]: This master's current role is: LEADER
I20260929 17:55:36.268366 14110 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 17:55:36.273964 14110 catalog_manager.cc:1347] Generated new cluster ID: 2860539585374904a1fcf2c8c03c6a81
I20260929 17:55:36.274039 14110 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 17:55:36.279817 14110 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 17:55:36.281036 14110 catalog_manager.cc:1504] Loading token signing keys...
I20260929 17:55:36.286104 14110 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5e0cc1a1df4444808156bdc323f6a303: Generated new TSK 0
I20260929 17:55:36.286840 14110 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 17:55:36.317736 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskup3Soe/build/debug/bin/kudu with pid 14028
2026-09-29T17:55:36Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1306 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 17:55:36 dist-test-slave-53nh krb5kdc[14123](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 17:55:36 dist-test-slave-53nh krb5kdc[14123](info): set up 2 sockets
Sep 29 17:55:36 dist-test-slave-53nh krb5kdc[14123](info): commencing operation
krb5kdc: starting...
W20260929 17:55:37.268930 392 mini_kdc.cc:121] Time spent starting KDC: real 0.932s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/krb5kdc/test-user.keytab.
Sep 29 17:55:37 dist-test-slave-53nh krb5kdc[14123](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704537, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T17:55:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T17:55:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 17:55:37.385910 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskup3Soe/build/debug/bin/kudu
/tmp/dist-test-taskup3Soe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44077
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44077
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:55:37.493247 14136 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:55:37.493505 14136 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:55:37.493546 14136 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:55:37.497514 14136 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 17:55:37.497571 14136 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:55:37.497589 14136 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:55:37.497604 14136 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 17:55:37.497618 14136 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 17:55:37.502840 14136 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44077
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44077
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14136
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:55:37.503865 14136 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:55:37.504834 14136 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:55:37.513056 14141 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:55:37.513049 14144 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:55:37.513132 14142 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:55:37.513145 14136 server_base.cc:1034] running on GCE node
I20260929 17:55:37.513804 14136 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:55:37.514846 14136 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:55:37.516002 14136 hybrid_clock.cc:648] HybridClock initialized: now 1790704537515981 us; error 38 us; skew 500 ppm
Sep 29 17:55:37 dist-test-slave-53nh krb5kdc[14123](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704537, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:55:37.518563 14136 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 17:55:37.519423 14136 webserver.cc:460] Webserver started at http://127.0.98.62:37917/ using document root <none> and password file <none>
I20260929 17:55:37.519938 14136 fs_manager.cc:359] Metadata directory not provided
I20260929 17:55:37.519997 14136 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:55:37.520188 14136 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:55:37.522176 14136 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/minicluster-data/master-0/data/instance:
uuid: "de3ecf1eeb38458faf6a908987c915f1"
format_stamp: "Formatted at 2026-09-29 17:55:37 on dist-test-slave-53nh"
server_key: "e54a357af30c51c020e49e8278319678"
server_key_iv: "447fc3407948338309cdb10f2b738ea7"
server_key_version: "encryptionkey@0"
I20260929 17:55:37.522699 14136 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/minicluster-data/master-0/wal/instance:
uuid: "de3ecf1eeb38458faf6a908987c915f1"
format_stamp: "Formatted at 2026-09-29 17:55:37 on dist-test-slave-53nh"
server_key: "e54a357af30c51c020e49e8278319678"
server_key_iv: "447fc3407948338309cdb10f2b738ea7"
server_key_version: "encryptionkey@0"
I20260929 17:55:37.526386 14136 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260929 17:55:37.528921 14151 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:55:37.530296 14136 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260929 17:55:37.530421 14136 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/minicluster-data/master-0/wal
uuid: "de3ecf1eeb38458faf6a908987c915f1"
format_stamp: "Formatted at 2026-09-29 17:55:37 on dist-test-slave-53nh"
server_key: "e54a357af30c51c020e49e8278319678"
server_key_iv: "447fc3407948338309cdb10f2b738ea7"
server_key_version: "encryptionkey@0"
I20260929 17:55:37.530530 14136 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:55:37.545957 14136 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:55:37.549312 14136 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:55:37.549501 14136 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:55:37.556324 14136 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44077
I20260929 17:55:37.556593 14203 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44077 every 8 connection(s)
I20260929 17:55:37.557376 14136 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/minicluster-data/master-0/data/info.pb
I20260929 17:55:37.560153 14204 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:55:37.561784 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskup3Soe/build/debug/bin/kudu as pid 14136
I20260929 17:55:37.561918 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/minicluster-data/master-0/wal/instance
I20260929 17:55:37.562198 392 external_mini_cluster.cc:1428] Setting key cf601f50d9267bea0aceb4a8521bbc52
I20260929 17:55:37.566712 14204 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de3ecf1eeb38458faf6a908987c915f1: Bootstrap starting.
I20260929 17:55:37.569413 14204 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P de3ecf1eeb38458faf6a908987c915f1: Neither blocks nor log segments found. Creating new log.
I20260929 17:55:37.570354 14204 log.cc:824] T 00000000000000000000000000000000 P de3ecf1eeb38458faf6a908987c915f1: Log is configured to *not* fsync() on all Append() calls
Sep 29 17:55:37 dist-test-slave-53nh krb5kdc[14123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704537, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 17:55:37.572613 14204 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de3ecf1eeb38458faf6a908987c915f1: No bootstrap required, opened a new log
I20260929 17:55:37.576423 14207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:55:37.563564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45938 (local address 127.0.98.62:44077)
0929 17:55:37.563967 (+ 403us) server_negotiation.cc:206] Beginning negotiation
0929 17:55:37.563983 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 17:55:37.564019 (+ 36us) server_negotiation.cc:407] Connection header received
0929 17:55:37.564656 (+ 637us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:55:37.564673 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:55:37.565043 (+ 370us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:55:37.565338 (+ 295us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:55:37.566182 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:55:37.567239 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:55:37.568409 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:55:37.568956 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:55:37.569178 (+ 222us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:55:37.571843 (+ 2665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:55:37.571848 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:55:37.571863 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:55:37.571893 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:55:37.573666 (+ 1773us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:55:37.574100 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:55:37.574105 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:55:37.574109 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:55:37.574224 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:55:37.574487 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:55:37.574490 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:55:37.574492 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:55:37.574900 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:55:37.575082 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0929 17:55:37.575374 (+ 292us) server_negotiation.cc:299] Negotiation successful
0929 17:55:37.575677 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":265,"thread_start_us":119,"threads_started":1}
I20260929 17:55:37.576313 14204 raft_consensus.cc:348] T 00000000000000000000000000000000 P de3ecf1eeb38458faf6a908987c915f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de3ecf1eeb38458faf6a908987c915f1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44077 } }
I20260929 17:55:37.576562 14204 raft_consensus.cc:374] T 00000000000000000000000000000000 P de3ecf1eeb38458faf6a908987c915f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:55:37.576612 14204 raft_consensus.cc:729] T 00000000000000000000000000000000 P de3ecf1eeb38458faf6a908987c915f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de3ecf1eeb38458faf6a908987c915f1, State: Initialized, Role: FOLLOWER
I20260929 17:55:37.577101 14204 consensus_queue.cc:260] T 00000000000000000000000000000000 P de3ecf1eeb38458faf6a908987c915f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de3ecf1eeb38458faf6a908987c915f1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44077 } }
I20260929 17:55:37.577244 14204 raft_consensus.cc:388] T 00000000000000000000000000000000 P de3ecf1eeb38458faf6a908987c915f1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 17:55:37.577312 14204 raft_consensus.cc:482] T 00000000000000000000000000000000 P de3ecf1eeb38458faf6a908987c915f1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 17:55:37.577410 14204 raft_consensus.cc:3037] T 00000000000000000000000000000000 P de3ecf1eeb38458faf6a908987c915f1 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:55:37.578367 14204 raft_consensus.cc:504] T 00000000000000000000000000000000 P de3ecf1eeb38458faf6a908987c915f1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de3ecf1eeb38458faf6a908987c915f1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44077 } }
I20260929 17:55:37.578722 14204 leader_election.cc:302] T 00000000000000000000000000000000 P de3ecf1eeb38458faf6a908987c915f1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: de3ecf1eeb38458faf6a908987c915f1; no voters:
I20260929 17:55:37.579028 14204 leader_election.cc:288] T 00000000000000000000000000000000 P de3ecf1eeb38458faf6a908987c915f1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 17:55:37.579123 14209 raft_consensus.cc:2781] T 00000000000000000000000000000000 P de3ecf1eeb38458faf6a908987c915f1 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 17:55:37.579332 14209 raft_consensus.cc:686] T 00000000000000000000000000000000 P de3ecf1eeb38458faf6a908987c915f1 [term 1 LEADER]: Becoming Leader. State: Replica: de3ecf1eeb38458faf6a908987c915f1, State: Running, Role: LEADER
I20260929 17:55:37.579646 14209 consensus_queue.cc:237] T 00000000000000000000000000000000 P de3ecf1eeb38458faf6a908987c915f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de3ecf1eeb38458faf6a908987c915f1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44077 } }
I20260929 17:55:37.580205 14204 sys_catalog.cc:564] T 00000000000000000000000000000000 P de3ecf1eeb38458faf6a908987c915f1 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 17:55:37.581310 14210 sys_catalog.cc:455] T 00000000000000000000000000000000 P de3ecf1eeb38458faf6a908987c915f1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "de3ecf1eeb38458faf6a908987c915f1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de3ecf1eeb38458faf6a908987c915f1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44077 } } }
I20260929 17:55:37.581480 14210 sys_catalog.cc:458] T 00000000000000000000000000000000 P de3ecf1eeb38458faf6a908987c915f1 [sys.catalog]: This master's current role is: LEADER
I20260929 17:55:37.581472 14211 sys_catalog.cc:455] T 00000000000000000000000000000000 P de3ecf1eeb38458faf6a908987c915f1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader de3ecf1eeb38458faf6a908987c915f1. Latest consensus state: current_term: 1 leader_uuid: "de3ecf1eeb38458faf6a908987c915f1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de3ecf1eeb38458faf6a908987c915f1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44077 } } }
I20260929 17:55:37.581583 14211 sys_catalog.cc:458] T 00000000000000000000000000000000 P de3ecf1eeb38458faf6a908987c915f1 [sys.catalog]: This master's current role is: LEADER
I20260929 17:55:37.581879 14219 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 17:55:37.585112 14219 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 17:55:37.590716 14219 catalog_manager.cc:1347] Generated new cluster ID: 374f9335722247a4aa9886591ecde9e8
I20260929 17:55:37.590790 14219 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 17:55:37.600888 14219 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 17:55:37.601915 14219 catalog_manager.cc:1504] Loading token signing keys...
I20260929 17:55:37.609964 14219 catalog_manager.cc:5868] T 00000000000000000000000000000000 P de3ecf1eeb38458faf6a908987c915f1: Generated new TSK 0
I20260929 17:55:37.610752 14219 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260929 17:55:37.637926 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskup3Soe/build/debug/bin/kudu
/tmp/dist-test-taskup3Soe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44077
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:55:37.748739 14230 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:55:37.749015 14230 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:55:37.749058 14230 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:55:37.752930 14230 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:55:37.752978 14230 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:55:37.753078 14230 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 17:55:37.758147 14230 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44077
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14230
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:55:37.759099 14230 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:55:37.760048 14230 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:55:37.769188 14235 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:55:37.769340 14230 server_base.cc:1034] running on GCE node
W20260929 17:55:37.769188 14238 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:55:37.769588 14236 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:55:37.769989 14230 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:55:37.770596 14230 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:55:37.771780 14230 hybrid_clock.cc:648] HybridClock initialized: now 1790704537771761 us; error 29 us; skew 500 ppm
Sep 29 17:55:37 dist-test-slave-53nh krb5kdc[14123](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704537, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:55:37.774537 14230 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260929 17:55:37.775468 14230 webserver.cc:460] Webserver started at http://127.0.98.1:37677/ using document root <none> and password file <none>
I20260929 17:55:37.776006 14230 fs_manager.cc:359] Metadata directory not provided
I20260929 17:55:37.776072 14230 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:55:37.776300 14230 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:55:37.778457 14230 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/minicluster-data/ts-0/data/instance:
uuid: "40fc602f689f47a2afb4b5ddf158d470"
format_stamp: "Formatted at 2026-09-29 17:55:37 on dist-test-slave-53nh"
server_key: "33beeb6c59ac0ce99be2d097572343fc"
server_key_iv: "fdb9495491df392eba099ab69555576f"
server_key_version: "encryptionkey@0"
I20260929 17:55:37.779026 14230 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/minicluster-data/ts-0/wal/instance:
uuid: "40fc602f689f47a2afb4b5ddf158d470"
format_stamp: "Formatted at 2026-09-29 17:55:37 on dist-test-slave-53nh"
server_key: "33beeb6c59ac0ce99be2d097572343fc"
server_key_iv: "fdb9495491df392eba099ab69555576f"
server_key_version: "encryptionkey@0"
I20260929 17:55:37.783104 14230 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 17:55:37.785640 14245 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:55:37.786872 14230 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 17:55:37.786986 14230 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/minicluster-data/ts-0/wal
uuid: "40fc602f689f47a2afb4b5ddf158d470"
format_stamp: "Formatted at 2026-09-29 17:55:37 on dist-test-slave-53nh"
server_key: "33beeb6c59ac0ce99be2d097572343fc"
server_key_iv: "fdb9495491df392eba099ab69555576f"
server_key_version: "encryptionkey@0"
I20260929 17:55:37.787091 14230 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:55:37.806258 14230 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:55:37.809988 14230 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:55:37.810209 14230 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:55:37.810834 14230 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 17:55:37.811699 14230 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:55:37.811745 14230 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:55:37.811789 14230 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:55:37.811818 14230 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 17:55:37.820385 14230 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37811
I20260929 17:55:37.820408 14358 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37811 every 8 connection(s)
I20260929 17:55:37.821290 14230 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/minicluster-data/ts-0/data/info.pb
I20260929 17:55:37.824290 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskup3Soe/build/debug/bin/kudu as pid 14230
I20260929 17:55:37.824421 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/minicluster-data/ts-0/wal/instance
I20260929 17:55:37.824733 392 external_mini_cluster.cc:1428] Setting key 1994c146738626c3b1c8fabd7d0969d6
Sep 29 17:55:37 dist-test-slave-53nh krb5kdc[14123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704537, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260929 17:55:37.835278 14207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:55:37.822946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43139 (local address 127.0.98.62:44077)
0929 17:55:37.823084 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0929 17:55:37.823088 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:55:37.823944 (+ 856us) server_negotiation.cc:407] Connection header received
0929 17:55:37.825452 (+ 1508us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:55:37.825481 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:55:37.825537 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:55:37.825614 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:55:37.827064 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:55:37.827674 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:55:37.828797 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:55:37.829089 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:55:37.829182 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:55:37.831756 (+ 2574us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:55:37.831759 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:55:37.831761 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:55:37.831784 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:55:37.833223 (+ 1439us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:55:37.833715 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:55:37.833719 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:55:37.833720 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:55:37.833795 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:55:37.834136 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:55:37.834140 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:55:37.834142 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:55:37.834313 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:55:37.834435 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0929 17:55:37.834958 (+ 523us) server_negotiation.cc:299] Negotiation successful
0929 17:55:37.835087 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260929 17:55:37.836141 14361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:55:37.823359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44077 (local address 127.0.98.1:43139)
0929 17:55:37.823795 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0929 17:55:37.823839 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 17:55:37.825250 (+ 1411us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:55:37.825791 (+ 541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:55:37.825799 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:55:37.826233 (+ 434us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:55:37.826897 (+ 664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:55:37.826915 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:55:37.827791 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:55:37.827794 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:55:37.828681 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:55:37.828690 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:55:37.829227 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:55:37.829231 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:55:37.829391 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:55:37.830039 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:55:37.830054 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:55:37.831598 (+ 1544us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:55:37.833337 (+ 1739us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:55:37.833343 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:55:37.833358 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:55:37.833623 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:55:37.833908 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:55:37.833911 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:55:37.833913 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:55:37.834041 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:55:37.834423 (+ 382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:55:37.834429 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:55:37.834712 (+ 283us) client_negotiation.cc:769] Sending connection context
0929 17:55:37.834887 (+ 175us) client_negotiation.cc:240] Negotiation successful
0929 17:55:37.835211 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":111,"threads_started":1}
I20260929 17:55:37.837452 14359 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44077
I20260929 17:55:37.837697 14359 heartbeater.cc:464] Registering TS with master...
I20260929 17:55:37.838182 14359 heartbeater.cc:511] Master 127.0.98.62:44077 requested a full tablet report, sending...
I20260929 17:55:37.839803 14168 ts_manager.cc:194] Registered new tserver with Master: 40fc602f689f47a2afb4b5ddf158d470 (127.0.98.1:37811)
I20260929 17:55:37.840950 14168 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:43139
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260929 17:55:37.851560 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskup3Soe/build/debug/bin/kudu
/tmp/dist-test-taskup3Soe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44077
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:55:37.960840 14364 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:55:37.961148 14364 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:55:37.961223 14364 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:55:37.965260 14364 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:55:37.965338 14364 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:55:37.965416 14364 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 17:55:37.970656 14364 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44077
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14364
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:55:37.971673 14364 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:55:37.972668 14364 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:55:37.982069 14369 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:55:37.982069 14370 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:55:37.982374 14372 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:55:37.982365 14364 server_base.cc:1034] running on GCE node
I20260929 17:55:37.982839 14364 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:55:37.983460 14364 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:55:37.984629 14364 hybrid_clock.cc:648] HybridClock initialized: now 1790704537984601 us; error 47 us; skew 500 ppm
Sep 29 17:55:37 dist-test-slave-53nh krb5kdc[14123](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704537, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:55:37.987254 14364 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260929 17:55:37.988101 14364 webserver.cc:460] Webserver started at http://127.0.98.2:43661/ using document root <none> and password file <none>
I20260929 17:55:37.988605 14364 fs_manager.cc:359] Metadata directory not provided
I20260929 17:55:37.988662 14364 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:55:37.988828 14364 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:55:37.990815 14364 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/minicluster-data/ts-1/data/instance:
uuid: "3f14c4f2dc464bcb93ec6832e5ca1767"
format_stamp: "Formatted at 2026-09-29 17:55:37 on dist-test-slave-53nh"
server_key: "8621cc99ae06ddd2899c6618480993fb"
server_key_iv: "cf2b8eef85248132087c75353cfef9b9"
server_key_version: "encryptionkey@0"
I20260929 17:55:37.991351 14364 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/minicluster-data/ts-1/wal/instance:
uuid: "3f14c4f2dc464bcb93ec6832e5ca1767"
format_stamp: "Formatted at 2026-09-29 17:55:37 on dist-test-slave-53nh"
server_key: "8621cc99ae06ddd2899c6618480993fb"
server_key_iv: "cf2b8eef85248132087c75353cfef9b9"
server_key_version: "encryptionkey@0"
I20260929 17:55:37.994864 14364 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 17:55:37.997092 14379 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:55:37.998311 14364 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 17:55:37.998405 14364 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/minicluster-data/ts-1/wal
uuid: "3f14c4f2dc464bcb93ec6832e5ca1767"
format_stamp: "Formatted at 2026-09-29 17:55:37 on dist-test-slave-53nh"
server_key: "8621cc99ae06ddd2899c6618480993fb"
server_key_iv: "cf2b8eef85248132087c75353cfef9b9"
server_key_version: "encryptionkey@0"
I20260929 17:55:37.998522 14364 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:55:38.019891 14364 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:55:38.023010 14364 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:55:38.023173 14364 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:55:38.023764 14364 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 17:55:38.024557 14364 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:55:38.024600 14364 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:55:38.024638 14364 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:55:38.024665 14364 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 17:55:38.032938 14364 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35183
I20260929 17:55:38.032964 14492 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35183 every 8 connection(s)
I20260929 17:55:38.033877 14364 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/minicluster-data/ts-1/data/info.pb
I20260929 17:55:38.037444 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskup3Soe/build/debug/bin/kudu as pid 14364
I20260929 17:55:38.037551 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/minicluster-data/ts-1/wal/instance
I20260929 17:55:38.037801 392 external_mini_cluster.cc:1428] Setting key ac0be6b3842cf7f8a3b64c326223b9d1
Sep 29 17:55:38 dist-test-slave-53nh krb5kdc[14123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704537, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260929 17:55:38.051990 14207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:55:38.035561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47677 (local address 127.0.98.62:44077)
0929 17:55:38.035693 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0929 17:55:38.035698 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:55:38.036524 (+ 826us) server_negotiation.cc:407] Connection header received
0929 17:55:38.037774 (+ 1250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:55:38.037777 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:55:38.037835 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:55:38.037925 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:55:38.039175 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:55:38.039904 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:55:38.040994 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:55:38.041338 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:55:38.041449 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:55:38.043880 (+ 2431us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:55:38.043884 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:55:38.043886 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:55:38.043909 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:55:38.049646 (+ 5737us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:55:38.050186 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:55:38.050189 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:55:38.050192 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:55:38.050285 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:55:38.050616 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:55:38.050620 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:55:38.050621 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:55:38.050842 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:55:38.050933 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0929 17:55:38.051410 (+ 477us) server_negotiation.cc:299] Negotiation successful
0929 17:55:38.051551 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260929 17:55:38.052587 14495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:55:38.035968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44077 (local address 127.0.98.2:47677)
0929 17:55:38.036381 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0929 17:55:38.036423 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 17:55:38.037564 (+ 1141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:55:38.038084 (+ 520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:55:38.038090 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:55:38.038506 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:55:38.039025 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:55:38.039044 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:55:38.040006 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:55:38.040009 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:55:38.040883 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:55:38.040891 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:55:38.041483 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:55:38.041486 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:55:38.041621 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:55:38.042153 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:55:38.042168 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:55:38.043717 (+ 1549us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:55:38.049787 (+ 6070us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:55:38.049793 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:55:38.049808 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:55:38.050095 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:55:38.050381 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:55:38.050384 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:55:38.050385 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:55:38.050532 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:55:38.050935 (+ 403us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:55:38.050940 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:55:38.051190 (+ 250us) client_negotiation.cc:769] Sending connection context
0929 17:55:38.051344 (+ 154us) client_negotiation.cc:240] Negotiation successful
0929 17:55:38.051650 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":98,"threads_started":1}
I20260929 17:55:38.053783 14493 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44077
I20260929 17:55:38.054008 14493 heartbeater.cc:464] Registering TS with master...
I20260929 17:55:38.054471 14493 heartbeater.cc:511] Master 127.0.98.62:44077 requested a full tablet report, sending...
I20260929 17:55:38.055477 14168 ts_manager.cc:194] Registered new tserver with Master: 3f14c4f2dc464bcb93ec6832e5ca1767 (127.0.98.2:35183)
I20260929 17:55:38.056023 14168 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:47677
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260929 17:55:38.061944 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskup3Soe/build/debug/bin/kudu
/tmp/dist-test-taskup3Soe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44077
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:55:38.168219 14498 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:55:38.168474 14498 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:55:38.168515 14498 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:55:38.172487 14498 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:55:38.172552 14498 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:55:38.172609 14498 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 17:55:38.177837 14498 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44077
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14498
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:55:38.178998 14498 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:55:38.179944 14498 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:55:38.189412 14503 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:55:38.189409 14506 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:55:38.189441 14504 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:55:38.189479 14498 server_base.cc:1034] running on GCE node
I20260929 17:55:38.189986 14498 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:55:38.190595 14498 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:55:38.191800 14498 hybrid_clock.cc:648] HybridClock initialized: now 1790704538191780 us; error 30 us; skew 500 ppm
Sep 29 17:55:38 dist-test-slave-53nh krb5kdc[14123](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704538, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:55:38.194391 14498 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260929 17:55:38.195273 14498 webserver.cc:460] Webserver started at http://127.0.98.3:45201/ using document root <none> and password file <none>
I20260929 17:55:38.195789 14498 fs_manager.cc:359] Metadata directory not provided
I20260929 17:55:38.195852 14498 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:55:38.196077 14498 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:55:38.198107 14498 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/minicluster-data/ts-2/data/instance:
uuid: "a5ea02aa22e3423e947ffd890f843ee7"
format_stamp: "Formatted at 2026-09-29 17:55:38 on dist-test-slave-53nh"
server_key: "d7f9e48bc909ed527929a4329e9d2231"
server_key_iv: "13942ed9259de28f8779a6c5d76a60ba"
server_key_version: "encryptionkey@0"
I20260929 17:55:38.198640 14498 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/minicluster-data/ts-2/wal/instance:
uuid: "a5ea02aa22e3423e947ffd890f843ee7"
format_stamp: "Formatted at 2026-09-29 17:55:38 on dist-test-slave-53nh"
server_key: "d7f9e48bc909ed527929a4329e9d2231"
server_key_iv: "13942ed9259de28f8779a6c5d76a60ba"
server_key_version: "encryptionkey@0"
I20260929 17:55:38.202538 14498 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 17:55:38.205312 14513 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:55:38.206598 14498 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 17:55:38.206719 14498 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/minicluster-data/ts-2/wal
uuid: "a5ea02aa22e3423e947ffd890f843ee7"
format_stamp: "Formatted at 2026-09-29 17:55:38 on dist-test-slave-53nh"
server_key: "d7f9e48bc909ed527929a4329e9d2231"
server_key_iv: "13942ed9259de28f8779a6c5d76a60ba"
server_key_version: "encryptionkey@0"
I20260929 17:55:38.206874 14498 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:55:38.227527 14498 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:55:38.230610 14498 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:55:38.230767 14498 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:55:38.231309 14498 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 17:55:38.232115 14498 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:55:38.232163 14498 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:55:38.232201 14498 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:55:38.232220 14498 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 17:55:38.240829 14498 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39167
I20260929 17:55:38.240852 14626 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39167 every 8 connection(s)
I20260929 17:55:38.241726 14498 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/minicluster-data/ts-2/data/info.pb
I20260929 17:55:38.247699 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskup3Soe/build/debug/bin/kudu as pid 14498
I20260929 17:55:38.247793 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/minicluster-data/ts-2/wal/instance
I20260929 17:55:38.248031 392 external_mini_cluster.cc:1428] Setting key fdd3cea1e323c77853038e18b4b7081b
Sep 29 17:55:38 dist-test-slave-53nh krb5kdc[14123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704538, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 17:55:38.254817 14207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:55:38.243493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58035 (local address 127.0.98.62:44077)
0929 17:55:38.243619 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0929 17:55:38.243622 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 17:55:38.244479 (+ 857us) server_negotiation.cc:407] Connection header received
0929 17:55:38.245736 (+ 1257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:55:38.245739 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:55:38.245798 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:55:38.245869 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:55:38.247092 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:55:38.247619 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:55:38.248901 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:55:38.249218 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:55:38.249315 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:55:38.251757 (+ 2442us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:55:38.251762 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:55:38.251765 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:55:38.251791 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:55:38.252902 (+ 1111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:55:38.253388 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:55:38.253391 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:55:38.253393 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:55:38.253478 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:55:38.253775 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:55:38.253778 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:55:38.253780 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:55:38.253963 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:55:38.254047 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0929 17:55:38.254531 (+ 484us) server_negotiation.cc:299] Negotiation successful
0929 17:55:38.254652 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 17:55:38.255606 14629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:55:38.243869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44077 (local address 127.0.98.3:58035)
0929 17:55:38.244310 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0929 17:55:38.244353 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 17:55:38.245522 (+ 1169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:55:38.246073 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:55:38.246081 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:55:38.246446 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:55:38.246943 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:55:38.246957 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:55:38.247735 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:55:38.247738 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:55:38.248780 (+ 1042us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:55:38.248790 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:55:38.249378 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:55:38.249382 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:55:38.249551 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:55:38.250199 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:55:38.250215 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:55:38.251592 (+ 1377us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:55:38.253034 (+ 1442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:55:38.253040 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:55:38.253056 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:55:38.253297 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:55:38.253556 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:55:38.253559 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:55:38.253561 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:55:38.253691 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:55:38.254054 (+ 363us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:55:38.254060 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:55:38.254303 (+ 243us) client_negotiation.cc:769] Sending connection context
0929 17:55:38.254469 (+ 166us) client_negotiation.cc:240] Negotiation successful
0929 17:55:38.254791 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":115,"threads_started":1}
I20260929 17:55:38.256726 14627 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44077
I20260929 17:55:38.256945 14627 heartbeater.cc:464] Registering TS with master...
I20260929 17:55:38.257421 14627 heartbeater.cc:511] Master 127.0.98.62:44077 requested a full tablet report, sending...
I20260929 17:55:38.258459 14168 ts_manager.cc:194] Registered new tserver with Master: a5ea02aa22e3423e947ffd890f843ee7 (127.0.98.3:39167)
I20260929 17:55:38.259008 14168 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:58035
I20260929 17:55:38.261624 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 17:55:38.270138 14207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:55:38.263406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45950 (local address 127.0.98.62:44077)
0929 17:55:38.263512 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0929 17:55:38.263516 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:55:38.263728 (+ 212us) server_negotiation.cc:407] Connection header received
0929 17:55:38.263831 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:55:38.263834 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:55:38.263873 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:55:38.263956 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:55:38.264570 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:55:38.265124 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:55:38.265939 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:55:38.266158 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:55:38.266242 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:55:38.267472 (+ 1230us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:55:38.267476 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:55:38.267479 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:55:38.267504 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:55:38.268632 (+ 1128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:55:38.268991 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:55:38.268994 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:55:38.269020 (+ 26us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:55:38.269098 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:55:38.269327 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:55:38.269330 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:55:38.269332 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:55:38.269544 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:55:38.269659 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0929 17:55:38.269870 (+ 211us) server_negotiation.cc:299] Negotiation successful
0929 17:55:38.269989 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260929 17:55:38.272748 14168 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45950:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 17:55:38.275934 14168 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 17:55:38.290498 14639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:55:38.284036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39167 (local address 127.0.0.1:45760)
0929 17:55:38.284817 (+ 781us) negotiation.cc:107] Waiting for socket to connect
0929 17:55:38.284885 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0929 17:55:38.285038 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:55:38.286205 (+ 1167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:55:38.286208 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:55:38.286222 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:55:38.286303 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:55:38.286308 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:55:38.287605 (+ 1297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:55:38.287608 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:55:38.288975 (+ 1367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:55:38.288985 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:55:38.289965 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:55:38.289968 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:55:38.290076 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:55:38.290114 (+ 38us) client_negotiation.cc:769] Sending connection context
0929 17:55:38.290220 (+ 106us) client_negotiation.cc:240] Negotiation successful
0929 17:55:38.290327 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":625,"mutex_wait_us":2,"thread_start_us":72,"threads_started":1}
I20260929 17:55:38.291003 14640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:55:38.284235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37811 (local address 127.0.0.1:42626)
0929 17:55:38.285465 (+ 1230us) negotiation.cc:107] Waiting for socket to connect
0929 17:55:38.285477 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 17:55:38.285568 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:55:38.286056 (+ 488us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:55:38.286061 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:55:38.286082 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:55:38.286188 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:55:38.286195 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:55:38.287640 (+ 1445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:55:38.287643 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:55:38.288892 (+ 1249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:55:38.288899 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:55:38.290680 (+ 1781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:55:38.290682 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:55:38.290744 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:55:38.290761 (+ 17us) client_negotiation.cc:769] Sending connection context
0929 17:55:38.290821 (+ 60us) client_negotiation.cc:240] Negotiation successful
0929 17:55:38.290883 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1175,"mutex_wait_us":36,"thread_start_us":120,"threads_started":1}
I20260929 17:55:38.291467 14643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:55:38.285149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42626 (local address 127.0.98.1:37811)
0929 17:55:38.285424 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0929 17:55:38.285435 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 17:55:38.285559 (+ 124us) server_negotiation.cc:407] Connection header received
0929 17:55:38.285676 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:55:38.285682 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:55:38.285893 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:55:38.286039 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:55:38.286300 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:55:38.287495 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:55:38.289015 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:55:38.289836 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:55:38.289949 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:55:38.290091 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0929 17:55:38.291231 (+ 1140us) server_negotiation.cc:299] Negotiation successful
0929 17:55:38.291340 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":90,"threads_started":1}
I20260929 17:55:38.290472 14641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:55:38.284191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45760 (local address 127.0.98.3:39167)
0929 17:55:38.285804 (+ 1613us) server_negotiation.cc:206] Beginning negotiation
0929 17:55:38.285816 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 17:55:38.285835 (+ 19us) server_negotiation.cc:407] Connection header received
0929 17:55:38.285915 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:55:38.285919 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:55:38.286085 (+ 166us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:55:38.286221 (+ 136us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:55:38.286368 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:55:38.287490 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:55:38.289120 (+ 1630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:55:38.289844 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:55:38.289949 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:55:38.290091 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0929 17:55:38.290252 (+ 161us) server_negotiation.cc:299] Negotiation successful
0929 17:55:38.290328 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1520,"thread_start_us":185,"threads_started":1}
I20260929 17:55:38.295753 14561 tablet_service.cc:1467] Processing CreateTablet for tablet cb67682f60bf4768b9a30f72b0e3d88b (DEFAULT_TABLE table=test-table [id=955b5bc23ea648a9bec39688b1b856a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 17:55:38.295760 14293 tablet_service.cc:1467] Processing CreateTablet for tablet cb67682f60bf4768b9a30f72b0e3d88b (DEFAULT_TABLE table=test-table [id=955b5bc23ea648a9bec39688b1b856a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 17:55:38.296737 14561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb67682f60bf4768b9a30f72b0e3d88b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:55:38.296868 14293 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb67682f60bf4768b9a30f72b0e3d88b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:55:38.304711 14644 tablet_bootstrap.cc:492] T cb67682f60bf4768b9a30f72b0e3d88b P a5ea02aa22e3423e947ffd890f843ee7: Bootstrap starting.
I20260929 17:55:38.305562 14645 tablet_bootstrap.cc:492] T cb67682f60bf4768b9a30f72b0e3d88b P 40fc602f689f47a2afb4b5ddf158d470: Bootstrap starting.
I20260929 17:55:38.307461 14644 tablet_bootstrap.cc:654] T cb67682f60bf4768b9a30f72b0e3d88b P a5ea02aa22e3423e947ffd890f843ee7: Neither blocks nor log segments found. Creating new log.
I20260929 17:55:38.307636 14645 tablet_bootstrap.cc:654] T cb67682f60bf4768b9a30f72b0e3d88b P 40fc602f689f47a2afb4b5ddf158d470: Neither blocks nor log segments found. Creating new log.
I20260929 17:55:38.308151 14642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:55:38.284900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44916 (local address 127.0.98.2:35183)
0929 17:55:38.285186 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0929 17:55:38.285197 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 17:55:38.287828 (+ 2631us) server_negotiation.cc:407] Connection header received
0929 17:55:38.287928 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:55:38.287933 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:55:38.288140 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:55:38.288293 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:55:38.288476 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:55:38.289414 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:55:38.290560 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:55:38.291331 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:55:38.291435 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:55:38.307789 (+ 16354us) server_negotiation.cc:1035] Waiting for connection context
0929 17:55:38.307859 (+ 70us) server_negotiation.cc:299] Negotiation successful
0929 17:55:38.307979 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":187,"thread_start_us":458,"threads_started":1}
I20260929 17:55:38.308480 14638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:55:38.284273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35183 (local address 127.0.0.1:44916)
0929 17:55:38.287716 (+ 3443us) negotiation.cc:107] Waiting for socket to connect
0929 17:55:38.287729 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 17:55:38.287812 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:55:38.288285 (+ 473us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:55:38.288289 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:55:38.288307 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:55:38.288386 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:55:38.288391 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:55:38.289522 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:55:38.289525 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:55:38.290400 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:55:38.290472 (+ 72us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:55:38.301083 (+ 10611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:55:38.301088 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:55:38.301209 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:55:38.301241 (+ 32us) client_negotiation.cc:769] Sending connection context
0929 17:55:38.301336 (+ 95us) client_negotiation.cc:240] Negotiation successful
0929 17:55:38.308339 (+ 7003us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3378,"thread_start_us":85,"threads_started":1}
I20260929 17:55:38.308442 14644 log.cc:824] T cb67682f60bf4768b9a30f72b0e3d88b P a5ea02aa22e3423e947ffd890f843ee7: Log is configured to *not* fsync() on all Append() calls
I20260929 17:55:38.309202 14645 log.cc:824] T cb67682f60bf4768b9a30f72b0e3d88b P 40fc602f689f47a2afb4b5ddf158d470: Log is configured to *not* fsync() on all Append() calls
I20260929 17:55:38.311343 14645 tablet_bootstrap.cc:492] T cb67682f60bf4768b9a30f72b0e3d88b P 40fc602f689f47a2afb4b5ddf158d470: No bootstrap required, opened a new log
I20260929 17:55:38.311542 14645 ts_tablet_manager.cc:1397] T cb67682f60bf4768b9a30f72b0e3d88b P 40fc602f689f47a2afb4b5ddf158d470: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260929 17:55:38.311575 14427 tablet_service.cc:1467] Processing CreateTablet for tablet cb67682f60bf4768b9a30f72b0e3d88b (DEFAULT_TABLE table=test-table [id=955b5bc23ea648a9bec39688b1b856a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 17:55:38.312624 14427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb67682f60bf4768b9a30f72b0e3d88b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:55:38.312660 14644 tablet_bootstrap.cc:492] T cb67682f60bf4768b9a30f72b0e3d88b P a5ea02aa22e3423e947ffd890f843ee7: No bootstrap required, opened a new log
I20260929 17:55:38.312830 14644 ts_tablet_manager.cc:1397] T cb67682f60bf4768b9a30f72b0e3d88b P a5ea02aa22e3423e947ffd890f843ee7: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.000s
I20260929 17:55:38.315259 14645 raft_consensus.cc:348] T cb67682f60bf4768b9a30f72b0e3d88b P 40fc602f689f47a2afb4b5ddf158d470 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5ea02aa22e3423e947ffd890f843ee7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39167 } } peers { permanent_uuid: "40fc602f689f47a2afb4b5ddf158d470" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37811 } } peers { permanent_uuid: "3f14c4f2dc464bcb93ec6832e5ca1767" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35183 } }
I20260929 17:55:38.315526 14645 raft_consensus.cc:374] T cb67682f60bf4768b9a30f72b0e3d88b P 40fc602f689f47a2afb4b5ddf158d470 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:55:38.315572 14645 raft_consensus.cc:729] T cb67682f60bf4768b9a30f72b0e3d88b P 40fc602f689f47a2afb4b5ddf158d470 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40fc602f689f47a2afb4b5ddf158d470, State: Initialized, Role: FOLLOWER
I20260929 17:55:38.316426 14645 consensus_queue.cc:260] T cb67682f60bf4768b9a30f72b0e3d88b P 40fc602f689f47a2afb4b5ddf158d470 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5ea02aa22e3423e947ffd890f843ee7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39167 } } peers { permanent_uuid: "40fc602f689f47a2afb4b5ddf158d470" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37811 } } peers { permanent_uuid: "3f14c4f2dc464bcb93ec6832e5ca1767" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35183 } }
I20260929 17:55:38.316658 14644 raft_consensus.cc:348] T cb67682f60bf4768b9a30f72b0e3d88b P a5ea02aa22e3423e947ffd890f843ee7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5ea02aa22e3423e947ffd890f843ee7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39167 } } peers { permanent_uuid: "40fc602f689f47a2afb4b5ddf158d470" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37811 } } peers { permanent_uuid: "3f14c4f2dc464bcb93ec6832e5ca1767" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35183 } }
I20260929 17:55:38.316926 14644 raft_consensus.cc:374] T cb67682f60bf4768b9a30f72b0e3d88b P a5ea02aa22e3423e947ffd890f843ee7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:55:38.316980 14644 raft_consensus.cc:729] T cb67682f60bf4768b9a30f72b0e3d88b P a5ea02aa22e3423e947ffd890f843ee7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5ea02aa22e3423e947ffd890f843ee7, State: Initialized, Role: FOLLOWER
I20260929 17:55:38.317456 14644 consensus_queue.cc:260] T cb67682f60bf4768b9a30f72b0e3d88b P a5ea02aa22e3423e947ffd890f843ee7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5ea02aa22e3423e947ffd890f843ee7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39167 } } peers { permanent_uuid: "40fc602f689f47a2afb4b5ddf158d470" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37811 } } peers { permanent_uuid: "3f14c4f2dc464bcb93ec6832e5ca1767" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35183 } }
I20260929 17:55:38.317756 14645 ts_tablet_manager.cc:1428] T cb67682f60bf4768b9a30f72b0e3d88b P 40fc602f689f47a2afb4b5ddf158d470: Time spent starting tablet: real 0.006s user 0.005s sys 0.001s
I20260929 17:55:38.318130 14359 heartbeater.cc:503] Master 127.0.98.62:44077 was elected leader, sending a full tablet report...
I20260929 17:55:38.318370 14644 ts_tablet_manager.cc:1428] T cb67682f60bf4768b9a30f72b0e3d88b P a5ea02aa22e3423e947ffd890f843ee7: Time spent starting tablet: real 0.005s user 0.003s sys 0.004s
I20260929 17:55:38.318722 14627 heartbeater.cc:503] Master 127.0.98.62:44077 was elected leader, sending a full tablet report...
I20260929 17:55:38.321132 14650 tablet_bootstrap.cc:492] T cb67682f60bf4768b9a30f72b0e3d88b P 3f14c4f2dc464bcb93ec6832e5ca1767: Bootstrap starting.
W20260929 17:55:38.322530 14360 tablet.cc:2369] T cb67682f60bf4768b9a30f72b0e3d88b P 40fc602f689f47a2afb4b5ddf158d470: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 17:55:38.323837 14650 tablet_bootstrap.cc:654] T cb67682f60bf4768b9a30f72b0e3d88b P 3f14c4f2dc464bcb93ec6832e5ca1767: Neither blocks nor log segments found. Creating new log.
I20260929 17:55:38.324599 14650 log.cc:824] T cb67682f60bf4768b9a30f72b0e3d88b P 3f14c4f2dc464bcb93ec6832e5ca1767: Log is configured to *not* fsync() on all Append() calls
I20260929 17:55:38.326272 14650 tablet_bootstrap.cc:492] T cb67682f60bf4768b9a30f72b0e3d88b P 3f14c4f2dc464bcb93ec6832e5ca1767: No bootstrap required, opened a new log
I20260929 17:55:38.326452 14650 ts_tablet_manager.cc:1397] T cb67682f60bf4768b9a30f72b0e3d88b P 3f14c4f2dc464bcb93ec6832e5ca1767: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 17:55:38.329515 14650 raft_consensus.cc:348] T cb67682f60bf4768b9a30f72b0e3d88b P 3f14c4f2dc464bcb93ec6832e5ca1767 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5ea02aa22e3423e947ffd890f843ee7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39167 } } peers { permanent_uuid: "40fc602f689f47a2afb4b5ddf158d470" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37811 } } peers { permanent_uuid: "3f14c4f2dc464bcb93ec6832e5ca1767" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35183 } }
I20260929 17:55:38.329735 14650 raft_consensus.cc:374] T cb67682f60bf4768b9a30f72b0e3d88b P 3f14c4f2dc464bcb93ec6832e5ca1767 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:55:38.329782 14650 raft_consensus.cc:729] T cb67682f60bf4768b9a30f72b0e3d88b P 3f14c4f2dc464bcb93ec6832e5ca1767 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f14c4f2dc464bcb93ec6832e5ca1767, State: Initialized, Role: FOLLOWER
I20260929 17:55:38.330181 14650 consensus_queue.cc:260] T cb67682f60bf4768b9a30f72b0e3d88b P 3f14c4f2dc464bcb93ec6832e5ca1767 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5ea02aa22e3423e947ffd890f843ee7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39167 } } peers { permanent_uuid: "40fc602f689f47a2afb4b5ddf158d470" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37811 } } peers { permanent_uuid: "3f14c4f2dc464bcb93ec6832e5ca1767" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35183 } }
I20260929 17:55:38.330842 14493 heartbeater.cc:503] Master 127.0.98.62:44077 was elected leader, sending a full tablet report...
I20260929 17:55:38.331066 14650 ts_tablet_manager.cc:1428] T cb67682f60bf4768b9a30f72b0e3d88b P 3f14c4f2dc464bcb93ec6832e5ca1767: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260929 17:55:38.485448 14648 raft_consensus.cc:482] T cb67682f60bf4768b9a30f72b0e3d88b P 40fc602f689f47a2afb4b5ddf158d470 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 17:55:38.485677 14648 raft_consensus.cc:504] T cb67682f60bf4768b9a30f72b0e3d88b P 40fc602f689f47a2afb4b5ddf158d470 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5ea02aa22e3423e947ffd890f843ee7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39167 } } peers { permanent_uuid: "40fc602f689f47a2afb4b5ddf158d470" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37811 } } peers { permanent_uuid: "3f14c4f2dc464bcb93ec6832e5ca1767" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35183 } }
I20260929 17:55:38.486799 14648 leader_election.cc:288] T cb67682f60bf4768b9a30f72b0e3d88b P 40fc602f689f47a2afb4b5ddf158d470 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a5ea02aa22e3423e947ffd890f843ee7 (127.0.98.3:39167), 3f14c4f2dc464bcb93ec6832e5ca1767 (127.0.98.2:35183)
I20260929 17:55:38.490767 14642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:55:38.487116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51225 (local address 127.0.98.2:35183)
0929 17:55:38.487251 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0929 17:55:38.487255 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:55:38.487511 (+ 256us) server_negotiation.cc:407] Connection header received
0929 17:55:38.487622 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:55:38.487625 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:55:38.487676 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:55:38.487742 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:55:38.488018 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:55:38.488554 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:55:38.489775 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:55:38.490179 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:55:38.490259 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:55:38.490313 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0929 17:55:38.490553 (+ 240us) server_negotiation.cc:299] Negotiation successful
0929 17:55:38.490638 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 17:55:38.490800 14653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:55:38.487017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39167 (local address 127.0.98.1:35207)
0929 17:55:38.487503 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0929 17:55:38.487518 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 17:55:38.487623 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:55:38.487860 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:55:38.487862 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:55:38.487875 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:55:38.487991 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:55:38.487998 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:55:38.488670 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:55:38.488673 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:55:38.489647 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:55:38.489656 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:55:38.490422 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:55:38.490424 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:55:38.490522 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:55:38.490544 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 17:55:38.490616 (+ 72us) client_negotiation.cc:240] Negotiation successful
0929 17:55:38.490679 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":423,"mutex_wait_us":197,"thread_start_us":82,"threads_started":1}
I20260929 17:55:38.490823 14641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:55:38.487138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35207 (local address 127.0.98.3:39167)
0929 17:55:38.487249 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0929 17:55:38.487254 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:55:38.487615 (+ 361us) server_negotiation.cc:407] Connection header received
0929 17:55:38.487712 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:55:38.487714 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:55:38.487765 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:55:38.487839 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:55:38.488094 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:55:38.488566 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:55:38.489790 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:55:38.490283 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:55:38.490411 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:55:38.490473 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0929 17:55:38.490644 (+ 171us) server_negotiation.cc:299] Negotiation successful
0929 17:55:38.490717 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 17:55:38.490824 14654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:55:38.487017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35183 (local address 127.0.98.1:51225)
0929 17:55:38.487416 (+ 399us) negotiation.cc:107] Waiting for socket to connect
0929 17:55:38.487429 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 17:55:38.487525 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:55:38.487761 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:55:38.487764 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:55:38.487778 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:55:38.487901 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:55:38.487908 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:55:38.488663 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:55:38.488666 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:55:38.489643 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:55:38.489655 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:55:38.490290 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:55:38.490293 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:55:38.490422 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:55:38.490445 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 17:55:38.490526 (+ 81us) client_negotiation.cc:240] Negotiation successful
0929 17:55:38.490607 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":312,"mutex_wait_us":88,"thread_start_us":115,"threads_started":1}
I20260929 17:55:38.491626 14447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb67682f60bf4768b9a30f72b0e3d88b" candidate_uuid: "40fc602f689f47a2afb4b5ddf158d470" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f14c4f2dc464bcb93ec6832e5ca1767" is_pre_election: true
I20260929 17:55:38.491694 14581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb67682f60bf4768b9a30f72b0e3d88b" candidate_uuid: "40fc602f689f47a2afb4b5ddf158d470" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5ea02aa22e3423e947ffd890f843ee7" is_pre_election: true
I20260929 17:55:38.491950 14581 raft_consensus.cc:2454] T cb67682f60bf4768b9a30f72b0e3d88b P a5ea02aa22e3423e947ffd890f843ee7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 40fc602f689f47a2afb4b5ddf158d470 in term 0.
I20260929 17:55:38.491950 14447 raft_consensus.cc:2454] T cb67682f60bf4768b9a30f72b0e3d88b P 3f14c4f2dc464bcb93ec6832e5ca1767 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 40fc602f689f47a2afb4b5ddf158d470 in term 0.
I20260929 17:55:38.492417 14248 leader_election.cc:302] T cb67682f60bf4768b9a30f72b0e3d88b P 40fc602f689f47a2afb4b5ddf158d470 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 40fc602f689f47a2afb4b5ddf158d470, a5ea02aa22e3423e947ffd890f843ee7; no voters:
W20260929 17:55:38.492602 14628 tablet.cc:2369] T cb67682f60bf4768b9a30f72b0e3d88b P a5ea02aa22e3423e947ffd890f843ee7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 17:55:38.492754 14648 raft_consensus.cc:2781] T cb67682f60bf4768b9a30f72b0e3d88b P 40fc602f689f47a2afb4b5ddf158d470 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 17:55:38.492849 14648 raft_consensus.cc:482] T cb67682f60bf4768b9a30f72b0e3d88b P 40fc602f689f47a2afb4b5ddf158d470 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 17:55:38.492897 14648 raft_consensus.cc:3037] T cb67682f60bf4768b9a30f72b0e3d88b P 40fc602f689f47a2afb4b5ddf158d470 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:55:38.493947 14648 raft_consensus.cc:504] T cb67682f60bf4768b9a30f72b0e3d88b P 40fc602f689f47a2afb4b5ddf158d470 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5ea02aa22e3423e947ffd890f843ee7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39167 } } peers { permanent_uuid: "40fc602f689f47a2afb4b5ddf158d470" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37811 } } peers { permanent_uuid: "3f14c4f2dc464bcb93ec6832e5ca1767" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35183 } }
I20260929 17:55:38.494282 14648 leader_election.cc:288] T cb67682f60bf4768b9a30f72b0e3d88b P 40fc602f689f47a2afb4b5ddf158d470 [CANDIDATE]: Term 1 election: Requested vote from peers a5ea02aa22e3423e947ffd890f843ee7 (127.0.98.3:39167), 3f14c4f2dc464bcb93ec6832e5ca1767 (127.0.98.2:35183)
I20260929 17:55:38.494649 14447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb67682f60bf4768b9a30f72b0e3d88b" candidate_uuid: "40fc602f689f47a2afb4b5ddf158d470" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f14c4f2dc464bcb93ec6832e5ca1767"
I20260929 17:55:38.494632 14581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb67682f60bf4768b9a30f72b0e3d88b" candidate_uuid: "40fc602f689f47a2afb4b5ddf158d470" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5ea02aa22e3423e947ffd890f843ee7"
I20260929 17:55:38.494761 14581 raft_consensus.cc:3037] T cb67682f60bf4768b9a30f72b0e3d88b P a5ea02aa22e3423e947ffd890f843ee7 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:55:38.494768 14447 raft_consensus.cc:3037] T cb67682f60bf4768b9a30f72b0e3d88b P 3f14c4f2dc464bcb93ec6832e5ca1767 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:55:38.495770 14447 raft_consensus.cc:2454] T cb67682f60bf4768b9a30f72b0e3d88b P 3f14c4f2dc464bcb93ec6832e5ca1767 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 40fc602f689f47a2afb4b5ddf158d470 in term 1.
I20260929 17:55:38.495785 14581 raft_consensus.cc:2454] T cb67682f60bf4768b9a30f72b0e3d88b P a5ea02aa22e3423e947ffd890f843ee7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 40fc602f689f47a2afb4b5ddf158d470 in term 1.
I20260929 17:55:38.496058 14246 leader_election.cc:302] T cb67682f60bf4768b9a30f72b0e3d88b P 40fc602f689f47a2afb4b5ddf158d470 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f14c4f2dc464bcb93ec6832e5ca1767, 40fc602f689f47a2afb4b5ddf158d470; no voters:
I20260929 17:55:38.496220 14648 raft_consensus.cc:2781] T cb67682f60bf4768b9a30f72b0e3d88b P 40fc602f689f47a2afb4b5ddf158d470 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 17:55:38.496443 14648 raft_consensus.cc:686] T cb67682f60bf4768b9a30f72b0e3d88b P 40fc602f689f47a2afb4b5ddf158d470 [term 1 LEADER]: Becoming Leader. State: Replica: 40fc602f689f47a2afb4b5ddf158d470, State: Running, Role: LEADER
I20260929 17:55:38.496793 14648 consensus_queue.cc:237] T cb67682f60bf4768b9a30f72b0e3d88b P 40fc602f689f47a2afb4b5ddf158d470 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5ea02aa22e3423e947ffd890f843ee7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39167 } } peers { permanent_uuid: "40fc602f689f47a2afb4b5ddf158d470" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37811 } } peers { permanent_uuid: "3f14c4f2dc464bcb93ec6832e5ca1767" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35183 } }
I20260929 17:55:38.499935 14168 catalog_manager.cc:5495] T cb67682f60bf4768b9a30f72b0e3d88b P 40fc602f689f47a2afb4b5ddf158d470 reported cstate change: term changed from 0 to 1, leader changed from <none> to 40fc602f689f47a2afb4b5ddf158d470 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "40fc602f689f47a2afb4b5ddf158d470" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5ea02aa22e3423e947ffd890f843ee7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39167 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "40fc602f689f47a2afb4b5ddf158d470" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37811 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3f14c4f2dc464bcb93ec6832e5ca1767" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35183 } health_report { overall_health: UNKNOWN } } }
W20260929 17:55:38.535027 14494 tablet.cc:2369] T cb67682f60bf4768b9a30f72b0e3d88b P 3f14c4f2dc464bcb93ec6832e5ca1767: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 17:55:38.539359 14643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:55:38.536327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51776 (local address 127.0.98.1:37811)
0929 17:55:38.536447 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0929 17:55:38.536450 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 17:55:38.536463 (+ 13us) server_negotiation.cc:407] Connection header received
0929 17:55:38.536535 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:55:38.536537 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:55:38.536591 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:55:38.536675 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 17:55:38.536859 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:55:38.537385 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:55:38.538169 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:55:38.538367 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:55:38.538436 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:55:38.538653 (+ 217us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 17:55:38.539063 (+ 410us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 17:55:38.539149 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0929 17:55:38.539175 (+ 26us) server_negotiation.cc:299] Negotiation successful
0929 17:55:38.539223 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 17:55:38.545213 14447 raft_consensus.cc:1261] T cb67682f60bf4768b9a30f72b0e3d88b P 3f14c4f2dc464bcb93ec6832e5ca1767 [term 1 FOLLOWER]: Refusing update from remote peer 40fc602f689f47a2afb4b5ddf158d470: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 17:55:38.545212 14581 raft_consensus.cc:1261] T cb67682f60bf4768b9a30f72b0e3d88b P a5ea02aa22e3423e947ffd890f843ee7 [term 1 FOLLOWER]: Refusing update from remote peer 40fc602f689f47a2afb4b5ddf158d470: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 17:55:38.545921 14648 consensus_queue.cc:1035] T cb67682f60bf4768b9a30f72b0e3d88b P 40fc602f689f47a2afb4b5ddf158d470 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5ea02aa22e3423e947ffd890f843ee7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39167 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 17:55:38.546128 14655 consensus_queue.cc:1035] T cb67682f60bf4768b9a30f72b0e3d88b P 40fc602f689f47a2afb4b5ddf158d470 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f14c4f2dc464bcb93ec6832e5ca1767" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 17:55:38.554020 14660 mvcc.cc:205] Tried to move back new op lower bound from 7334725789873524736 to 7334725789685768192. Current Snapshot: MvccSnapshot[applied={T|T < 7334725789873524736}]
I20260929 17:55:38.562611 14168 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45950:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 17:55:38.562799 14168 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45950:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 17:55:38.565102 14168 catalog_manager.cc:5782] T 00000000000000000000000000000000 P de3ecf1eeb38458faf6a908987c915f1: Sending DeleteTablet for 3 replicas of tablet cb67682f60bf4768b9a30f72b0e3d88b
I20260929 17:55:38.565899 14561 tablet_service.cc:1514] Processing DeleteTablet for tablet cb67682f60bf4768b9a30f72b0e3d88b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 17:55:38 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45760
I20260929 17:55:38.566115 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskup3Soe/build/debug/bin/kudu with pid 14230
I20260929 17:55:38.566098 14427 tablet_service.cc:1514] Processing DeleteTablet for tablet cb67682f60bf4768b9a30f72b0e3d88b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 17:55:38 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44916
I20260929 17:55:38.566897 14669 tablet_replica.cc:330] T cb67682f60bf4768b9a30f72b0e3d88b P 3f14c4f2dc464bcb93ec6832e5ca1767: stopping tablet replica
I20260929 17:55:38.567232 14669 raft_consensus.cc:2229] T cb67682f60bf4768b9a30f72b0e3d88b P 3f14c4f2dc464bcb93ec6832e5ca1767 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 17:55:38.567545 14669 raft_consensus.cc:2258] T cb67682f60bf4768b9a30f72b0e3d88b P 3f14c4f2dc464bcb93ec6832e5ca1767 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 17:55:38.569041 14669 ts_tablet_manager.cc:1905] T cb67682f60bf4768b9a30f72b0e3d88b P 3f14c4f2dc464bcb93ec6832e5ca1767: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 17:55:38.571327 14669 ts_tablet_manager.cc:1918] T cb67682f60bf4768b9a30f72b0e3d88b P 3f14c4f2dc464bcb93ec6832e5ca1767: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 17:55:38.571421 14669 log.cc:1196] T cb67682f60bf4768b9a30f72b0e3d88b P 3f14c4f2dc464bcb93ec6832e5ca1767: Deleting WAL directory at /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/minicluster-data/ts-1/wal/wals/cb67682f60bf4768b9a30f72b0e3d88b
W20260929 17:55:38.571403 14153 connection.cc:535] client connection to 127.0.98.1:37811 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260929 17:55:38.571589 14153 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260929 17:55:38.571691 14669 ts_tablet_manager.cc:1939] T cb67682f60bf4768b9a30f72b0e3d88b P 3f14c4f2dc464bcb93ec6832e5ca1767: Deleting consensus metadata
W20260929 17:55:38.572619 14638 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:55:38.572334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37811 (local address 127.0.0.1:51782)
0929 17:55:38.572464 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0929 17:55:38.572534 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37811: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":47}
W20260929 17:55:38.572755 14153 catalog_manager.cc:4584] TS 40fc602f689f47a2afb4b5ddf158d470 (127.0.98.1:37811): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet cb67682f60bf4768b9a30f72b0e3d88b: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37811: connect: Connection reset by peer (error 104)
I20260929 17:55:38.572791 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskup3Soe/build/debug/bin/kudu with pid 14364
I20260929 17:55:38.573355 14152 catalog_manager.cc:4841] TS 3f14c4f2dc464bcb93ec6832e5ca1767 (127.0.98.2:35183): tablet cb67682f60bf4768b9a30f72b0e3d88b (table test-table [id=955b5bc23ea648a9bec39688b1b856a7]) successfully deleted
I20260929 17:55:38.577294 14668 tablet_replica.cc:330] T cb67682f60bf4768b9a30f72b0e3d88b P a5ea02aa22e3423e947ffd890f843ee7: stopping tablet replica
I20260929 17:55:38.577705 14668 raft_consensus.cc:2229] T cb67682f60bf4768b9a30f72b0e3d88b P a5ea02aa22e3423e947ffd890f843ee7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 17:55:38.578112 14668 raft_consensus.cc:2258] T cb67682f60bf4768b9a30f72b0e3d88b P a5ea02aa22e3423e947ffd890f843ee7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 17:55:38.579447 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskup3Soe/build/debug/bin/kudu with pid 14498
I20260929 17:55:38.585059 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskup3Soe/build/debug/bin/kudu with pid 14136
I20260929 17:55:38.590417 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskup3Soe/build/debug/bin/kudu
/tmp/dist-test-taskup3Soe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44077
--webserver_interface=127.0.98.62
--webserver_port=37917
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44077
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:55:38.699672 14671 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:55:38.699978 14671 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:55:38.700019 14671 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:55:38.704283 14671 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 17:55:38.704358 14671 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:55:38.704376 14671 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:55:38.704391 14671 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 17:55:38.704406 14671 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 17:55:38.709905 14671 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44077
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44077
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=37917
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14671
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:55:38.710954 14671 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:55:38.711907 14671 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:55:38.724866 14678 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:55:38.724869 14680 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:55:38.724870 14677 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:55:38.725243 14671 server_base.cc:1034] running on GCE node
I20260929 17:55:38.725936 14671 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:55:38.727173 14671 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:55:38.728355 14671 hybrid_clock.cc:648] HybridClock initialized: now 1790704538728317 us; error 55 us; skew 500 ppm
Sep 29 17:55:38 dist-test-slave-53nh krb5kdc[14123](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704538, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:55:38.731249 14671 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 17:55:38.732241 14671 webserver.cc:460] Webserver started at http://127.0.98.62:37917/ using document root <none> and password file <none>
I20260929 17:55:38.732846 14671 fs_manager.cc:359] Metadata directory not provided
I20260929 17:55:38.732929 14671 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:55:38.737069 14671 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 17:55:38.739164 14687 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:55:38.740425 14671 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 17:55:38.740540 14671 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/minicluster-data/master-0/wal
uuid: "de3ecf1eeb38458faf6a908987c915f1"
format_stamp: "Formatted at 2026-09-29 17:55:37 on dist-test-slave-53nh"
server_key: "e54a357af30c51c020e49e8278319678"
server_key_iv: "447fc3407948338309cdb10f2b738ea7"
server_key_version: "encryptionkey@0"
I20260929 17:55:38.741178 14671 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:55:38.752063 14671 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:55:38.754750 14671 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:55:38.754917 14671 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:55:38.761546 14671 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44077
I20260929 17:55:38.761876 14739 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44077 every 8 connection(s)
I20260929 17:55:38.762663 14671 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/minicluster-data/master-0/data/info.pb
I20260929 17:55:38.766206 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskup3Soe/build/debug/bin/kudu as pid 14671
I20260929 17:55:38.766595 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskup3Soe/build/debug/bin/kudu
/tmp/dist-test-taskup3Soe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:37811
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:44077
--webserver_port=37677
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 17:55:38.767216 14740 sys_catalog.cc:263] Verifying existing consensus state
I20260929 17:55:38.769415 14740 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de3ecf1eeb38458faf6a908987c915f1: Bootstrap starting.
I20260929 17:55:38.786183 14740 log.cc:824] T 00000000000000000000000000000000 P de3ecf1eeb38458faf6a908987c915f1: Log is configured to *not* fsync() on all Append() calls
I20260929 17:55:38.794710 14740 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de3ecf1eeb38458faf6a908987c915f1: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 17:55:38.795047 14740 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de3ecf1eeb38458faf6a908987c915f1: Bootstrap complete.
I20260929 17:55:38.798822 14740 raft_consensus.cc:348] T 00000000000000000000000000000000 P de3ecf1eeb38458faf6a908987c915f1 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de3ecf1eeb38458faf6a908987c915f1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44077 } }
I20260929 17:55:38.799216 14740 raft_consensus.cc:729] T 00000000000000000000000000000000 P de3ecf1eeb38458faf6a908987c915f1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: de3ecf1eeb38458faf6a908987c915f1, State: Initialized, Role: FOLLOWER
I20260929 17:55:38.799687 14740 consensus_queue.cc:260] T 00000000000000000000000000000000 P de3ecf1eeb38458faf6a908987c915f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de3ecf1eeb38458faf6a908987c915f1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44077 } }
I20260929 17:55:38.799811 14740 raft_consensus.cc:388] T 00000000000000000000000000000000 P de3ecf1eeb38458faf6a908987c915f1 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 17:55:38.799881 14740 raft_consensus.cc:482] T 00000000000000000000000000000000 P de3ecf1eeb38458faf6a908987c915f1 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 17:55:38.799973 14740 raft_consensus.cc:3037] T 00000000000000000000000000000000 P de3ecf1eeb38458faf6a908987c915f1 [term 1 FOLLOWER]: Advancing to term 2
I20260929 17:55:38.801911 14740 raft_consensus.cc:504] T 00000000000000000000000000000000 P de3ecf1eeb38458faf6a908987c915f1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de3ecf1eeb38458faf6a908987c915f1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44077 } }
I20260929 17:55:38.802220 14740 leader_election.cc:302] T 00000000000000000000000000000000 P de3ecf1eeb38458faf6a908987c915f1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: de3ecf1eeb38458faf6a908987c915f1; no voters:
I20260929 17:55:38.802531 14740 leader_election.cc:288] T 00000000000000000000000000000000 P de3ecf1eeb38458faf6a908987c915f1 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260929 17:55:38.802655 14744 raft_consensus.cc:2781] T 00000000000000000000000000000000 P de3ecf1eeb38458faf6a908987c915f1 [term 2 FOLLOWER]: Leader election won for term 2
I20260929 17:55:38.802852 14744 raft_consensus.cc:686] T 00000000000000000000000000000000 P de3ecf1eeb38458faf6a908987c915f1 [term 2 LEADER]: Becoming Leader. State: Replica: de3ecf1eeb38458faf6a908987c915f1, State: Running, Role: LEADER
I20260929 17:55:38.803212 14744 consensus_queue.cc:237] T 00000000000000000000000000000000 P de3ecf1eeb38458faf6a908987c915f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de3ecf1eeb38458faf6a908987c915f1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44077 } }
I20260929 17:55:38.803776 14740 sys_catalog.cc:564] T 00000000000000000000000000000000 P de3ecf1eeb38458faf6a908987c915f1 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 17:55:38.804631 14745 sys_catalog.cc:455] T 00000000000000000000000000000000 P de3ecf1eeb38458faf6a908987c915f1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "de3ecf1eeb38458faf6a908987c915f1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de3ecf1eeb38458faf6a908987c915f1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44077 } } }
I20260929 17:55:38.804765 14745 sys_catalog.cc:458] T 00000000000000000000000000000000 P de3ecf1eeb38458faf6a908987c915f1 [sys.catalog]: This master's current role is: LEADER
I20260929 17:55:38.805028 14746 sys_catalog.cc:455] T 00000000000000000000000000000000 P de3ecf1eeb38458faf6a908987c915f1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader de3ecf1eeb38458faf6a908987c915f1. Latest consensus state: current_term: 2 leader_uuid: "de3ecf1eeb38458faf6a908987c915f1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de3ecf1eeb38458faf6a908987c915f1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44077 } } }
I20260929 17:55:38.805105 14746 sys_catalog.cc:458] T 00000000000000000000000000000000 P de3ecf1eeb38458faf6a908987c915f1 [sys.catalog]: This master's current role is: LEADER
I20260929 17:55:38.805438 14753 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 17:55:38.810761 14753 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 17:55:38.811621 14753 catalog_manager.cc:1259] Loaded cluster ID: 374f9335722247a4aa9886591ecde9e8
I20260929 17:55:38.811681 14753 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 17:55:38.813027 14753 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260929 17:55:38.817524 14753 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260929 17:55:38.822806 14753 catalog_manager.cc:1504] Loading token signing keys...
I20260929 17:55:38.823580 14753 catalog_manager.cc:5879] T 00000000000000000000000000000000 P de3ecf1eeb38458faf6a908987c915f1: Loaded TSK: 0
I20260929 17:55:38.824783 14753 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 17:55:38.882874 14742 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:55:38.883136 14742 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:55:38.883180 14742 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:55:38.887275 14742 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:55:38.887348 14742 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:55:38.887408 14742 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 17:55:38.892755 14742 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:37811
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=37677
--tserver_master_addrs=127.0.98.62:44077
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14742
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:55:38.893814 14742 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:55:38.894848 14742 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:55:38.905786 14767 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:55:38.905719 14768 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:55:38.906065 14770 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:55:38.906246 14742 server_base.cc:1034] running on GCE node
I20260929 17:55:38.906672 14742 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:55:38.907311 14742 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:55:38.908507 14742 hybrid_clock.cc:648] HybridClock initialized: now 1790704538908473 us; error 52 us; skew 500 ppm
Sep 29 17:55:38 dist-test-slave-53nh krb5kdc[14123](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704538, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:55:38.911332 14742 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260929 17:55:38.912222 14742 webserver.cc:460] Webserver started at http://127.0.98.1:37677/ using document root <none> and password file <none>
I20260929 17:55:38.912766 14742 fs_manager.cc:359] Metadata directory not provided
I20260929 17:55:38.912838 14742 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:55:38.916923 14742 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260929 17:55:38.919435 14777 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:55:38.920944 14742 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 17:55:38.921108 14742 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/minicluster-data/ts-0/wal
uuid: "40fc602f689f47a2afb4b5ddf158d470"
format_stamp: "Formatted at 2026-09-29 17:55:37 on dist-test-slave-53nh"
server_key: "33beeb6c59ac0ce99be2d097572343fc"
server_key_iv: "fdb9495491df392eba099ab69555576f"
server_key_version: "encryptionkey@0"
I20260929 17:55:38.921682 14742 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:55:38.934593 14742 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:55:38.937908 14742 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:55:38.938112 14742 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:55:38.938707 14742 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 17:55:38.939774 14784 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260929 17:55:38.945218 14742 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260929 17:55:38.945298 14742 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260929 17:55:38.945366 14742 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260929 17:55:38.947091 14742 ts_tablet_manager.cc:610] Registered 1 tablets
I20260929 17:55:38.947144 14742 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260929 17:55:38.947489 14784 tablet_bootstrap.cc:492] T cb67682f60bf4768b9a30f72b0e3d88b P 40fc602f689f47a2afb4b5ddf158d470: Bootstrap starting.
I20260929 17:55:38.956750 14742 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37811
I20260929 17:55:38.957222 14891 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37811 every 8 connection(s)
I20260929 17:55:38.957844 14742 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/minicluster-data/ts-0/data/info.pb
I20260929 17:55:38.963256 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskup3Soe/build/debug/bin/kudu as pid 14742
I20260929 17:55:38.963660 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskup3Soe/build/debug/bin/kudu
/tmp/dist-test-taskup3Soe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:35183
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:44077
--webserver_port=43661
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 17:55:38.966807 14784 log.cc:824] T cb67682f60bf4768b9a30f72b0e3d88b P 40fc602f689f47a2afb4b5ddf158d470: Log is configured to *not* fsync() on all Append() calls
Sep 29 17:55:38 dist-test-slave-53nh krb5kdc[14123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704538, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 17:55:38.975196 14784 tablet_bootstrap.cc:492] T cb67682f60bf4768b9a30f72b0e3d88b P 40fc602f689f47a2afb4b5ddf158d470: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 17:55:38.975565 14784 tablet_bootstrap.cc:492] T cb67682f60bf4768b9a30f72b0e3d88b P 40fc602f689f47a2afb4b5ddf158d470: Bootstrap complete.
I20260929 17:55:38.976781 14784 ts_tablet_manager.cc:1397] T cb67682f60bf4768b9a30f72b0e3d88b P 40fc602f689f47a2afb4b5ddf158d470: Time spent bootstrapping tablet: real 0.030s user 0.018s sys 0.007s
I20260929 17:55:38.977480 14894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:55:38.959999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50027 (local address 127.0.98.62:44077)
0929 17:55:38.960456 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0929 17:55:38.960476 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0929 17:55:38.960907 (+ 431us) server_negotiation.cc:407] Connection header received
0929 17:55:38.962093 (+ 1186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:55:38.962105 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:55:38.962426 (+ 321us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:55:38.962738 (+ 312us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:55:38.964517 (+ 1779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:55:38.966127 (+ 1610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:55:38.967653 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:55:38.968312 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:55:38.968499 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:55:38.971589 (+ 3090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:55:38.971597 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:55:38.971617 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:55:38.971656 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:55:38.973841 (+ 2185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:55:38.974457 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:55:38.974464 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:55:38.974470 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:55:38.974626 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:55:38.974987 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:55:38.974990 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:55:38.974992 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:55:38.975454 (+ 462us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:55:38.975581 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0929 17:55:38.976249 (+ 668us) server_negotiation.cc:299] Negotiation successful
0929 17:55:38.976567 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329,"thread_start_us":110,"threads_started":1}
I20260929 17:55:38.977499 14895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:55:38.959883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44077 (local address 127.0.98.1:50027)
0929 17:55:38.960735 (+ 852us) negotiation.cc:107] Waiting for socket to connect
0929 17:55:38.960789 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0929 17:55:38.961676 (+ 887us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:55:38.962881 (+ 1205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:55:38.962894 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:55:38.963602 (+ 708us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:55:38.964293 (+ 691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:55:38.964318 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:55:38.966294 (+ 1976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:55:38.966297 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:55:38.967517 (+ 1220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:55:38.967529 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:55:38.968474 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:55:38.968478 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:55:38.968693 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:55:38.969413 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:55:38.969435 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:55:38.971328 (+ 1893us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:55:38.973980 (+ 2652us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:55:38.973986 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:55:38.973998 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:55:38.974338 (+ 340us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:55:38.974727 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:55:38.974730 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:55:38.974732 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:55:38.974897 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:55:38.975572 (+ 675us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:55:38.975578 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:55:38.975914 (+ 336us) client_negotiation.cc:769] Sending connection context
0929 17:55:38.976100 (+ 186us) client_negotiation.cc:240] Negotiation successful
0929 17:55:38.976438 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":645,"thread_start_us":148,"threads_started":1}
I20260929 17:55:38.979372 14784 raft_consensus.cc:348] T cb67682f60bf4768b9a30f72b0e3d88b P 40fc602f689f47a2afb4b5ddf158d470 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5ea02aa22e3423e947ffd890f843ee7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39167 } } peers { permanent_uuid: "40fc602f689f47a2afb4b5ddf158d470" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37811 } } peers { permanent_uuid: "3f14c4f2dc464bcb93ec6832e5ca1767" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35183 } }
I20260929 17:55:38.979768 14784 raft_consensus.cc:729] T cb67682f60bf4768b9a30f72b0e3d88b P 40fc602f689f47a2afb4b5ddf158d470 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40fc602f689f47a2afb4b5ddf158d470, State: Initialized, Role: FOLLOWER
I20260929 17:55:38.980224 14892 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44077
I20260929 17:55:38.980304 14784 consensus_queue.cc:260] T cb67682f60bf4768b9a30f72b0e3d88b P 40fc602f689f47a2afb4b5ddf158d470 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5ea02aa22e3423e947ffd890f843ee7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39167 } } peers { permanent_uuid: "40fc602f689f47a2afb4b5ddf158d470" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37811 } } peers { permanent_uuid: "3f14c4f2dc464bcb93ec6832e5ca1767" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35183 } }
I20260929 17:55:38.980466 14892 heartbeater.cc:464] Registering TS with master...
I20260929 17:55:38.981045 14892 heartbeater.cc:511] Master 127.0.98.62:44077 requested a full tablet report, sending...
I20260929 17:55:38.981320 14784 ts_tablet_manager.cc:1428] T cb67682f60bf4768b9a30f72b0e3d88b P 40fc602f689f47a2afb4b5ddf158d470: Time spent starting tablet: real 0.004s user 0.004s sys 0.004s
I20260929 17:55:38.983652 14704 ts_manager.cc:194] Registered new tserver with Master: 40fc602f689f47a2afb4b5ddf158d470 (127.0.98.1:37811)
I20260929 17:55:38.987082 14704 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:50027
I20260929 17:55:38.989494 14892 heartbeater.cc:503] Master 127.0.98.62:44077 was elected leader, sending a full tablet report...
Sep 29 17:55:38 dist-test-slave-53nh krb5kdc[14123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704538, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260929 17:55:38.997437 14899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:55:38.986504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37811 (local address 127.0.0.1:51790)
0929 17:55:38.987280 (+ 776us) negotiation.cc:107] Waiting for socket to connect
0929 17:55:38.987329 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 17:55:38.987486 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:55:38.988175 (+ 689us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:55:38.988180 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:55:38.988612 (+ 432us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:55:38.988740 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:55:38.988748 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:55:38.990011 (+ 1263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:55:38.990014 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:55:38.990668 (+ 654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:55:38.990678 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:55:38.991327 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:55:38.991345 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:55:38.991466 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:55:38.991986 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:55:38.992000 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:55:38.993526 (+ 1526us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:55:38.995421 (+ 1895us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:55:38.995431 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:55:38.995437 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:55:38.995669 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:55:38.996110 (+ 441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:55:38.996113 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:55:38.996115 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:55:38.996211 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:55:38.996704 (+ 493us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:55:38.996710 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:55:38.996878 (+ 168us) client_negotiation.cc:769] Sending connection context
0929 17:55:38.997026 (+ 148us) client_negotiation.cc:240] Negotiation successful
0929 17:55:38.997250 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":602,"thread_start_us":101,"threads_started":1}
I20260929 17:55:38.997463 14900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:55:38.986621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51790 (local address 127.0.98.1:37811)
0929 17:55:38.987007 (+ 386us) server_negotiation.cc:206] Beginning negotiation
0929 17:55:38.987019 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 17:55:38.987443 (+ 424us) server_negotiation.cc:407] Connection header received
0929 17:55:38.987661 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:55:38.987668 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:55:38.987871 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:55:38.987974 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:55:38.988880 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:55:38.989888 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:55:38.990780 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:55:38.991206 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:55:38.991313 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:55:38.993659 (+ 2346us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:55:38.993668 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:55:38.993679 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:55:38.993702 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:55:38.995287 (+ 1585us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:55:38.995827 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:55:38.995831 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:55:38.995833 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:55:38.995911 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:55:38.996305 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:55:38.996308 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:55:38.996311 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:55:38.996591 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:55:38.996711 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0929 17:55:38.997132 (+ 421us) server_negotiation.cc:299] Negotiation successful
0929 17:55:38.997323 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":297,"thread_start_us":109,"threads_started":1}
I20260929 17:55:38.998543 14826 tablet_service.cc:1514] Processing DeleteTablet for tablet cb67682f60bf4768b9a30f72b0e3d88b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 17:55:38 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51790
I20260929 17:55:38.999460 14901 tablet_replica.cc:330] stopping tablet replica
I20260929 17:55:38.999877 14901 raft_consensus.cc:2229] T cb67682f60bf4768b9a30f72b0e3d88b P 40fc602f689f47a2afb4b5ddf158d470 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 17:55:39.000173 14901 raft_consensus.cc:2258] T cb67682f60bf4768b9a30f72b0e3d88b P 40fc602f689f47a2afb4b5ddf158d470 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 17:55:39.001435 14901 ts_tablet_manager.cc:1905] T cb67682f60bf4768b9a30f72b0e3d88b P 40fc602f689f47a2afb4b5ddf158d470: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 17:55:39.006278 14901 ts_tablet_manager.cc:1918] T cb67682f60bf4768b9a30f72b0e3d88b P 40fc602f689f47a2afb4b5ddf158d470: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 17:55:39.006403 14901 log.cc:1196] T cb67682f60bf4768b9a30f72b0e3d88b P 40fc602f689f47a2afb4b5ddf158d470: Deleting WAL directory at /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/minicluster-data/ts-0/wal/wals/cb67682f60bf4768b9a30f72b0e3d88b
I20260929 17:55:39.006652 14901 ts_tablet_manager.cc:1939] T cb67682f60bf4768b9a30f72b0e3d88b P 40fc602f689f47a2afb4b5ddf158d470: Deleting consensus metadata
I20260929 17:55:39.008787 14689 catalog_manager.cc:4841] TS 40fc602f689f47a2afb4b5ddf158d470 (127.0.98.1:37811): tablet cb67682f60bf4768b9a30f72b0e3d88b (table test-table [id=955b5bc23ea648a9bec39688b1b856a7]) successfully deleted
W20260929 17:55:39.086158 14896 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:55:39.086453 14896 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:55:39.086501 14896 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:55:39.090600 14896 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:55:39.090670 14896 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:55:39.090741 14896 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 17:55:39.096009 14896 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:35183
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=43661
--tserver_master_addrs=127.0.98.62:44077
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14896
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:55:39.097043 14896 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:55:39.098021 14896 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:55:39.107841 14909 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:55:39.107870 14906 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:55:39.107911 14907 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:55:39.107910 14896 server_base.cc:1034] running on GCE node
I20260929 17:55:39.108534 14896 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:55:39.109251 14896 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:55:39.110448 14896 hybrid_clock.cc:648] HybridClock initialized: now 1790704539110408 us; error 34 us; skew 500 ppm
Sep 29 17:55:39 dist-test-slave-53nh krb5kdc[14123](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704539, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:55:39.113127 14896 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260929 17:55:39.114001 14896 webserver.cc:460] Webserver started at http://127.0.98.2:43661/ using document root <none> and password file <none>
I20260929 17:55:39.114533 14896 fs_manager.cc:359] Metadata directory not provided
I20260929 17:55:39.114612 14896 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:55:39.118399 14896 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260929 17:55:39.120414 14916 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:55:39.121714 14896 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 17:55:39.121821 14896 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/minicluster-data/ts-1/wal
uuid: "3f14c4f2dc464bcb93ec6832e5ca1767"
format_stamp: "Formatted at 2026-09-29 17:55:37 on dist-test-slave-53nh"
server_key: "8621cc99ae06ddd2899c6618480993fb"
server_key_iv: "cf2b8eef85248132087c75353cfef9b9"
server_key_version: "encryptionkey@0"
I20260929 17:55:39.122347 14896 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:55:39.135648 14896 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:55:39.138492 14896 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:55:39.138664 14896 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:55:39.139236 14896 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 17:55:39.140038 14896 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:55:39.140084 14896 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:55:39.140134 14896 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:55:39.140161 14896 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 17:55:39.148837 14896 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35183
I20260929 17:55:39.148877 15029 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35183 every 8 connection(s)
I20260929 17:55:39.149878 14896 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/minicluster-data/ts-1/data/info.pb
I20260929 17:55:39.150472 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskup3Soe/build/debug/bin/kudu as pid 14896
I20260929 17:55:39.150933 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskup3Soe/build/debug/bin/kudu
/tmp/dist-test-taskup3Soe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:39167
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:44077
--webserver_port=45201
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 17:55:39 dist-test-slave-53nh krb5kdc[14123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704539, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 17:55:39.166035 14894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:55:39.151928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50197 (local address 127.0.98.62:44077)
0929 17:55:39.152093 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0929 17:55:39.152098 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:55:39.153111 (+ 1013us) server_negotiation.cc:407] Connection header received
0929 17:55:39.154399 (+ 1288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:55:39.154402 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:55:39.154465 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:55:39.154551 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:55:39.156253 (+ 1702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:55:39.156788 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:55:39.158548 (+ 1760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:55:39.158790 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:55:39.158894 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:55:39.161909 (+ 3015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:55:39.161914 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:55:39.161916 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:55:39.161946 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:55:39.163759 (+ 1813us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:55:39.164271 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:55:39.164275 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:55:39.164277 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:55:39.164368 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:55:39.164779 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:55:39.164782 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:55:39.164784 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:55:39.165054 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:55:39.165146 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0929 17:55:39.165687 (+ 541us) server_negotiation.cc:299] Negotiation successful
0929 17:55:39.165835 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 17:55:39.166893 15033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:55:39.152435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44077 (local address 127.0.98.2:50197)
0929 17:55:39.152939 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0929 17:55:39.152982 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 17:55:39.154175 (+ 1193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:55:39.154803 (+ 628us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:55:39.154814 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:55:39.155326 (+ 512us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:55:39.156071 (+ 745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:55:39.156094 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:55:39.156914 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:55:39.156918 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:55:39.158389 (+ 1471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:55:39.158401 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:55:39.158953 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:55:39.158958 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:55:39.159233 (+ 275us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:55:39.159869 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:55:39.159891 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:55:39.161683 (+ 1792us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:55:39.163891 (+ 2208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:55:39.163896 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:55:39.163910 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:55:39.164174 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:55:39.164477 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:55:39.164480 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:55:39.164481 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:55:39.164657 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:55:39.165154 (+ 497us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:55:39.165161 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:55:39.165440 (+ 279us) client_negotiation.cc:769] Sending connection context
0929 17:55:39.165608 (+ 168us) client_negotiation.cc:240] Negotiation successful
0929 17:55:39.165961 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":147,"threads_started":1}
I20260929 17:55:39.168133 15030 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44077
I20260929 17:55:39.168372 15030 heartbeater.cc:464] Registering TS with master...
I20260929 17:55:39.168869 15030 heartbeater.cc:511] Master 127.0.98.62:44077 requested a full tablet report, sending...
I20260929 17:55:39.169996 14704 ts_manager.cc:194] Registered new tserver with Master: 3f14c4f2dc464bcb93ec6832e5ca1767 (127.0.98.2:35183)
I20260929 17:55:39.170773 14704 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:50197
W20260929 17:55:39.265686 15032 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:55:39.265972 15032 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:55:39.266016 15032 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:55:39.270193 15032 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:55:39.270273 15032 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:55:39.270339 15032 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 17:55:39.275769 15032 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:39167
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=45201
--tserver_master_addrs=127.0.98.62:44077
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15032
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:55:39.276885 15032 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:55:39.277920 15032 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:55:39.287345 15041 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:55:39.287345 15039 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:55:39.287345 15038 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:55:39.287488 15032 server_base.cc:1034] running on GCE node
I20260929 17:55:39.288125 15032 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:55:39.288794 15032 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:55:39.289976 15032 hybrid_clock.cc:648] HybridClock initialized: now 1790704539289946 us; error 47 us; skew 500 ppm
Sep 29 17:55:39 dist-test-slave-53nh krb5kdc[14123](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704539, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:55:39.292567 15032 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260929 17:55:39.293514 15032 webserver.cc:460] Webserver started at http://127.0.98.3:45201/ using document root <none> and password file <none>
I20260929 17:55:39.294055 15032 fs_manager.cc:359] Metadata directory not provided
I20260929 17:55:39.294139 15032 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:55:39.297938 15032 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260929 17:55:39.300050 15048 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:55:39.301473 15032 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 17:55:39.301577 15032 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/minicluster-data/ts-2/wal
uuid: "a5ea02aa22e3423e947ffd890f843ee7"
format_stamp: "Formatted at 2026-09-29 17:55:38 on dist-test-slave-53nh"
server_key: "d7f9e48bc909ed527929a4329e9d2231"
server_key_iv: "13942ed9259de28f8779a6c5d76a60ba"
server_key_version: "encryptionkey@0"
I20260929 17:55:39.302124 15032 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:55:39.311712 15032 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:55:39.314812 15032 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:55:39.315032 15032 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:55:39.315773 15032 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 17:55:39.317027 15055 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260929 17:55:39.321365 15032 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260929 17:55:39.321458 15032 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.001s sys 0.000s
I20260929 17:55:39.321528 15032 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260929 17:55:39.323141 15032 ts_tablet_manager.cc:610] Registered 1 tablets
I20260929 17:55:39.323190 15032 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260929 17:55:39.323500 15055 tablet_bootstrap.cc:492] T cb67682f60bf4768b9a30f72b0e3d88b P a5ea02aa22e3423e947ffd890f843ee7: Bootstrap starting.
I20260929 17:55:39.330125 15055 log.cc:824] T cb67682f60bf4768b9a30f72b0e3d88b P a5ea02aa22e3423e947ffd890f843ee7: Log is configured to *not* fsync() on all Append() calls
I20260929 17:55:39.332074 15032 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39167
I20260929 17:55:39.332165 15163 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39167 every 8 connection(s)
I20260929 17:55:39.332968 15032 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/minicluster-data/ts-2/data/info.pb
I20260929 17:55:39.337646 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskup3Soe/build/debug/bin/kudu as pid 15032
I20260929 17:55:39.339118 15055 tablet_bootstrap.cc:492] T cb67682f60bf4768b9a30f72b0e3d88b P a5ea02aa22e3423e947ffd890f843ee7: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 17:55:39.339515 15055 tablet_bootstrap.cc:492] T cb67682f60bf4768b9a30f72b0e3d88b P a5ea02aa22e3423e947ffd890f843ee7: Bootstrap complete.
I20260929 17:55:39.339869 15055 ts_tablet_manager.cc:1397] T cb67682f60bf4768b9a30f72b0e3d88b P a5ea02aa22e3423e947ffd890f843ee7: Time spent bootstrapping tablet: real 0.017s user 0.011s sys 0.004s
I20260929 17:55:39.343374 15055 raft_consensus.cc:348] T cb67682f60bf4768b9a30f72b0e3d88b P a5ea02aa22e3423e947ffd890f843ee7 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5ea02aa22e3423e947ffd890f843ee7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39167 } } peers { permanent_uuid: "40fc602f689f47a2afb4b5ddf158d470" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37811 } } peers { permanent_uuid: "3f14c4f2dc464bcb93ec6832e5ca1767" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35183 } }
I20260929 17:55:39.343849 15055 raft_consensus.cc:729] T cb67682f60bf4768b9a30f72b0e3d88b P a5ea02aa22e3423e947ffd890f843ee7 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5ea02aa22e3423e947ffd890f843ee7, State: Initialized, Role: FOLLOWER
Sep 29 17:55:39 dist-test-slave-53nh krb5kdc[14123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704539, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 17:55:39.344398 15055 consensus_queue.cc:260] T cb67682f60bf4768b9a30f72b0e3d88b P a5ea02aa22e3423e947ffd890f843ee7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5ea02aa22e3423e947ffd890f843ee7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39167 } } peers { permanent_uuid: "40fc602f689f47a2afb4b5ddf158d470" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37811 } } peers { permanent_uuid: "3f14c4f2dc464bcb93ec6832e5ca1767" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35183 } }
I20260929 17:55:39.345446 15055 ts_tablet_manager.cc:1428] T cb67682f60bf4768b9a30f72b0e3d88b P a5ea02aa22e3423e947ffd890f843ee7: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260929 17:55:39.347092 15168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:55:39.338217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45012 (local address 127.0.98.62:44077)
0929 17:55:39.339155 (+ 938us) server_negotiation.cc:206] Beginning negotiation
0929 17:55:39.339159 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:55:39.339520 (+ 361us) server_negotiation.cc:407] Connection header received
0929 17:55:39.339700 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:55:39.339704 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:55:39.339765 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:55:39.339837 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:55:39.340612 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:55:39.341171 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:55:39.342166 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:55:39.342440 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:55:39.342529 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:55:39.343929 (+ 1400us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:55:39.343933 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:55:39.343936 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:55:39.343961 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:55:39.345573 (+ 1612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:55:39.345974 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:55:39.345978 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:55:39.345980 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:55:39.346064 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:55:39.346308 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:55:39.346312 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:55:39.346314 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:55:39.346505 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:55:39.346712 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0929 17:55:39.346800 (+ 88us) server_negotiation.cc:299] Negotiation successful
0929 17:55:39.346920 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":862,"thread_start_us":69,"threads_started":1}
I20260929 17:55:39.348610 14894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:55:39.334895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40861 (local address 127.0.98.62:44077)
0929 17:55:39.335072 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0929 17:55:39.335076 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:55:39.335646 (+ 570us) server_negotiation.cc:407] Connection header received
0929 17:55:39.337280 (+ 1634us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:55:39.337283 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:55:39.337330 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:55:39.337421 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:55:39.339047 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:55:39.339738 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:55:39.341326 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:55:39.341644 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:55:39.341743 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:55:39.344966 (+ 3223us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:55:39.344970 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:55:39.344972 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:55:39.345035 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:55:39.346428 (+ 1393us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:55:39.347057 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:55:39.347060 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:55:39.347061 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:55:39.347139 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:55:39.347489 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:55:39.347492 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:55:39.347494 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:55:39.347694 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:55:39.347769 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0929 17:55:39.348346 (+ 577us) server_negotiation.cc:299] Negotiation successful
0929 17:55:39.348474 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260929 17:55:39.349493 15166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:55:39.335010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44077 (local address 127.0.98.3:40861)
0929 17:55:39.335479 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0929 17:55:39.335533 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0929 17:55:39.337068 (+ 1535us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:55:39.337567 (+ 499us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:55:39.337575 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:55:39.338025 (+ 450us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:55:39.338898 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:55:39.338917 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:55:39.339871 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:55:39.339875 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:55:39.341221 (+ 1346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:55:39.341231 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:55:39.341779 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:55:39.341783 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:55:39.341981 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:55:39.342670 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:55:39.342691 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:55:39.344538 (+ 1847us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:55:39.346528 (+ 1990us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:55:39.346535 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:55:39.346587 (+ 52us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:55:39.346932 (+ 345us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:55:39.347236 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:55:39.347239 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:55:39.347241 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:55:39.347409 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:55:39.347800 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:55:39.347807 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:55:39.348100 (+ 293us) client_negotiation.cc:769] Sending connection context
0929 17:55:39.348288 (+ 188us) client_negotiation.cc:240] Negotiation successful
0929 17:55:39.348547 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":94,"threads_started":1}
I20260929 17:55:39.350486 15164 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44077
I20260929 17:55:39.350703 15164 heartbeater.cc:464] Registering TS with master...
I20260929 17:55:39.351261 15164 heartbeater.cc:511] Master 127.0.98.62:44077 requested a full tablet report, sending...
I20260929 17:55:39.352946 14704 ts_manager.cc:194] Registered new tserver with Master: a5ea02aa22e3423e947ffd890f843ee7 (127.0.98.3:39167)
I20260929 17:55:39.354058 14704 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:40861
I20260929 17:55:39.356313 15164 heartbeater.cc:503] Master 127.0.98.62:44077 was elected leader, sending a full tablet report...
Sep 29 17:55:39 dist-test-slave-53nh krb5kdc[14123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704538, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
I20260929 17:55:39.360912 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 17:55:39.363445 14899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:55:39.353943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39167 (local address 127.0.0.1:45530)
0929 17:55:39.354099 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0929 17:55:39.354111 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 17:55:39.354189 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:55:39.354881 (+ 692us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:55:39.354883 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:55:39.355035 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:55:39.355119 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:55:39.355124 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:55:39.356446 (+ 1322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:55:39.356450 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:55:39.357212 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:55:39.357224 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:55:39.357891 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:55:39.357904 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:55:39.357975 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:55:39.358462 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:55:39.358473 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:55:39.359706 (+ 1233us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:55:39.361781 (+ 2075us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:55:39.361783 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:55:39.361784 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:55:39.361967 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:55:39.362326 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:55:39.362329 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:55:39.362331 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:55:39.362429 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:55:39.362922 (+ 493us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:55:39.362925 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:55:39.363055 (+ 130us) client_negotiation.cc:769] Sending connection context
0929 17:55:39.363142 (+ 87us) client_negotiation.cc:240] Negotiation successful
0929 17:55:39.363281 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97,"mutex_wait_us":18}
I20260929 17:55:39.363588 15170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:55:39.354066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45530 (local address 127.0.98.3:39167)
0929 17:55:39.354416 (+ 350us) server_negotiation.cc:206] Beginning negotiation
0929 17:55:39.354432 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 17:55:39.354450 (+ 18us) server_negotiation.cc:407] Connection header received
0929 17:55:39.354506 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:55:39.354510 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:55:39.354725 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:55:39.354871 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:55:39.355220 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:55:39.356301 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:55:39.357334 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:55:39.357792 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:55:39.357894 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:55:39.359879 (+ 1985us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:55:39.359887 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:55:39.359898 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:55:39.359925 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:55:39.361669 (+ 1744us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:55:39.362076 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:55:39.362083 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:55:39.362085 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:55:39.362190 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:55:39.362509 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:55:39.362513 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:55:39.362515 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:55:39.362819 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:55:39.362949 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0929 17:55:39.363243 (+ 294us) server_negotiation.cc:299] Negotiation successful
0929 17:55:39.363414 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":254,"thread_start_us":133,"threads_started":1}
I20260929 17:55:39.364614 15097 tablet_service.cc:1514] Processing DeleteTablet for tablet cb67682f60bf4768b9a30f72b0e3d88b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 17:55:38 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45530
I20260929 17:55:39.365376 14894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:55:39.361519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45026 (local address 127.0.98.62:44077)
0929 17:55:39.361647 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0929 17:55:39.361651 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:55:39.361828 (+ 177us) server_negotiation.cc:407] Connection header received
0929 17:55:39.361996 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:55:39.361998 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:55:39.362092 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:55:39.362187 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 17:55:39.362437 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:55:39.362998 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:55:39.363978 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:55:39.364184 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:55:39.364264 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:55:39.364570 (+ 306us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 17:55:39.364949 (+ 379us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 17:55:39.365087 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0929 17:55:39.365192 (+ 105us) server_negotiation.cc:299] Negotiation successful
0929 17:55:39.365247 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":19,"server-negotiator.queue_time_us":56}
I20260929 17:55:39.365581 15172 tablet_replica.cc:330] stopping tablet replica
I20260929 17:55:39.365921 15172 raft_consensus.cc:2229] T cb67682f60bf4768b9a30f72b0e3d88b P a5ea02aa22e3423e947ffd890f843ee7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 17:55:39.366205 15172 raft_consensus.cc:2258] T cb67682f60bf4768b9a30f72b0e3d88b P a5ea02aa22e3423e947ffd890f843ee7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 17:55:39.367271 15172 ts_tablet_manager.cc:1905] T cb67682f60bf4768b9a30f72b0e3d88b P a5ea02aa22e3423e947ffd890f843ee7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 17:55:39.366066 14704 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:45026:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 17:55:39.370218 14704 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 17:55:39.371886 15172 ts_tablet_manager.cc:1918] T cb67682f60bf4768b9a30f72b0e3d88b P a5ea02aa22e3423e947ffd890f843ee7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 17:55:39.371965 15172 log.cc:1196] T cb67682f60bf4768b9a30f72b0e3d88b P a5ea02aa22e3423e947ffd890f843ee7: Deleting WAL directory at /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/minicluster-data/ts-2/wal/wals/cb67682f60bf4768b9a30f72b0e3d88b
I20260929 17:55:39.372193 15172 ts_tablet_manager.cc:1939] T cb67682f60bf4768b9a30f72b0e3d88b P a5ea02aa22e3423e947ffd890f843ee7: Deleting consensus metadata
I20260929 17:55:39.373998 14690 catalog_manager.cc:4841] TS a5ea02aa22e3423e947ffd890f843ee7 (127.0.98.3:39167): tablet cb67682f60bf4768b9a30f72b0e3d88b (table test-table [id=955b5bc23ea648a9bec39688b1b856a7]) successfully deleted
I20260929 17:55:39.378026 15097 tablet_service.cc:1467] Processing CreateTablet for tablet 34cb3888e81d468884bb1e9936a65cba (DEFAULT_TABLE table=test-table [id=6bc0edaac07640729c421e8935be307b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 17:55:39.378393 14826 tablet_service.cc:1467] Processing CreateTablet for tablet 34cb3888e81d468884bb1e9936a65cba (DEFAULT_TABLE table=test-table [id=6bc0edaac07640729c421e8935be307b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 17:55:39.378551 15097 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34cb3888e81d468884bb1e9936a65cba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:55:39.378974 14826 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34cb3888e81d468884bb1e9936a65cba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:55:39.381083 15055 tablet_bootstrap.cc:492] T 34cb3888e81d468884bb1e9936a65cba P a5ea02aa22e3423e947ffd890f843ee7: Bootstrap starting.
I20260929 17:55:39.381536 14784 tablet_bootstrap.cc:492] T 34cb3888e81d468884bb1e9936a65cba P 40fc602f689f47a2afb4b5ddf158d470: Bootstrap starting.
I20260929 17:55:39.382014 15055 tablet_bootstrap.cc:654] T 34cb3888e81d468884bb1e9936a65cba P a5ea02aa22e3423e947ffd890f843ee7: Neither blocks nor log segments found. Creating new log.
I20260929 17:55:39.382292 14784 tablet_bootstrap.cc:654] T 34cb3888e81d468884bb1e9936a65cba P 40fc602f689f47a2afb4b5ddf158d470: Neither blocks nor log segments found. Creating new log.
I20260929 17:55:39.382987 14899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:55:39.377370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35183 (local address 127.0.0.1:48214)
0929 17:55:39.377475 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0929 17:55:39.377489 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 17:55:39.377553 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:55:39.378565 (+ 1012us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:55:39.378568 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:55:39.378588 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:55:39.378694 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:55:39.378701 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:55:39.380562 (+ 1861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:55:39.380565 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:55:39.381703 (+ 1138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:55:39.381713 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:55:39.382564 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:55:39.382567 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:55:39.382655 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:55:39.382701 (+ 46us) client_negotiation.cc:769] Sending connection context
0929 17:55:39.382780 (+ 79us) client_negotiation.cc:240] Negotiation successful
0929 17:55:39.382860 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260929 17:55:39.383040 15176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:55:39.377693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48214 (local address 127.0.98.2:35183)
0929 17:55:39.378039 (+ 346us) server_negotiation.cc:206] Beginning negotiation
0929 17:55:39.378051 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 17:55:39.378074 (+ 23us) server_negotiation.cc:407] Connection header received
0929 17:55:39.378150 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:55:39.378156 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:55:39.378399 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:55:39.378569 (+ 170us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:55:39.378797 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:55:39.380420 (+ 1623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:55:39.381860 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:55:39.382452 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:55:39.382552 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:55:39.382704 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0929 17:55:39.382809 (+ 105us) server_negotiation.cc:299] Negotiation successful
0929 17:55:39.382904 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":251,"thread_start_us":153,"threads_started":1}
I20260929 17:55:39.383668 14784 tablet_bootstrap.cc:492] T 34cb3888e81d468884bb1e9936a65cba P 40fc602f689f47a2afb4b5ddf158d470: No bootstrap required, opened a new log
I20260929 17:55:39.383754 14784 ts_tablet_manager.cc:1397] T 34cb3888e81d468884bb1e9936a65cba P 40fc602f689f47a2afb4b5ddf158d470: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260929 17:55:39.384073 15055 tablet_bootstrap.cc:492] T 34cb3888e81d468884bb1e9936a65cba P a5ea02aa22e3423e947ffd890f843ee7: No bootstrap required, opened a new log
I20260929 17:55:39.384152 15055 ts_tablet_manager.cc:1397] T 34cb3888e81d468884bb1e9936a65cba P a5ea02aa22e3423e947ffd890f843ee7: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260929 17:55:39.384187 14784 raft_consensus.cc:348] T 34cb3888e81d468884bb1e9936a65cba P 40fc602f689f47a2afb4b5ddf158d470 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f14c4f2dc464bcb93ec6832e5ca1767" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35183 } } peers { permanent_uuid: "a5ea02aa22e3423e947ffd890f843ee7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39167 } } peers { permanent_uuid: "40fc602f689f47a2afb4b5ddf158d470" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37811 } }
I20260929 17:55:39.384316 14784 raft_consensus.cc:374] T 34cb3888e81d468884bb1e9936a65cba P 40fc602f689f47a2afb4b5ddf158d470 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:55:39.384352 14784 raft_consensus.cc:729] T 34cb3888e81d468884bb1e9936a65cba P 40fc602f689f47a2afb4b5ddf158d470 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40fc602f689f47a2afb4b5ddf158d470, State: Initialized, Role: FOLLOWER
I20260929 17:55:39.384461 14784 consensus_queue.cc:260] T 34cb3888e81d468884bb1e9936a65cba P 40fc602f689f47a2afb4b5ddf158d470 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f14c4f2dc464bcb93ec6832e5ca1767" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35183 } } peers { permanent_uuid: "a5ea02aa22e3423e947ffd890f843ee7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39167 } } peers { permanent_uuid: "40fc602f689f47a2afb4b5ddf158d470" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37811 } }
I20260929 17:55:39.384547 15055 raft_consensus.cc:348] T 34cb3888e81d468884bb1e9936a65cba P a5ea02aa22e3423e947ffd890f843ee7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f14c4f2dc464bcb93ec6832e5ca1767" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35183 } } peers { permanent_uuid: "a5ea02aa22e3423e947ffd890f843ee7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39167 } } peers { permanent_uuid: "40fc602f689f47a2afb4b5ddf158d470" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37811 } }
I20260929 17:55:39.384646 15055 raft_consensus.cc:374] T 34cb3888e81d468884bb1e9936a65cba P a5ea02aa22e3423e947ffd890f843ee7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:55:39.384677 15055 raft_consensus.cc:729] T 34cb3888e81d468884bb1e9936a65cba P a5ea02aa22e3423e947ffd890f843ee7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5ea02aa22e3423e947ffd890f843ee7, State: Initialized, Role: FOLLOWER
I20260929 17:55:39.384693 14784 ts_tablet_manager.cc:1428] T 34cb3888e81d468884bb1e9936a65cba P 40fc602f689f47a2afb4b5ddf158d470: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260929 17:55:39.384783 15055 consensus_queue.cc:260] T 34cb3888e81d468884bb1e9936a65cba P a5ea02aa22e3423e947ffd890f843ee7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f14c4f2dc464bcb93ec6832e5ca1767" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35183 } } peers { permanent_uuid: "a5ea02aa22e3423e947ffd890f843ee7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39167 } } peers { permanent_uuid: "40fc602f689f47a2afb4b5ddf158d470" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37811 } }
I20260929 17:55:39.385036 15055 ts_tablet_manager.cc:1428] T 34cb3888e81d468884bb1e9936a65cba P a5ea02aa22e3423e947ffd890f843ee7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 17:55:39.386099 14964 tablet_service.cc:1467] Processing CreateTablet for tablet 34cb3888e81d468884bb1e9936a65cba (DEFAULT_TABLE table=test-table [id=6bc0edaac07640729c421e8935be307b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 17:55:39.387171 14964 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34cb3888e81d468884bb1e9936a65cba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:55:39.393606 15179 tablet_bootstrap.cc:492] T 34cb3888e81d468884bb1e9936a65cba P 3f14c4f2dc464bcb93ec6832e5ca1767: Bootstrap starting.
I20260929 17:55:39.395798 15179 tablet_bootstrap.cc:654] T 34cb3888e81d468884bb1e9936a65cba P 3f14c4f2dc464bcb93ec6832e5ca1767: Neither blocks nor log segments found. Creating new log.
I20260929 17:55:39.396625 15179 log.cc:824] T 34cb3888e81d468884bb1e9936a65cba P 3f14c4f2dc464bcb93ec6832e5ca1767: Log is configured to *not* fsync() on all Append() calls
I20260929 17:55:39.398386 15179 tablet_bootstrap.cc:492] T 34cb3888e81d468884bb1e9936a65cba P 3f14c4f2dc464bcb93ec6832e5ca1767: No bootstrap required, opened a new log
I20260929 17:55:39.398550 15179 ts_tablet_manager.cc:1397] T 34cb3888e81d468884bb1e9936a65cba P 3f14c4f2dc464bcb93ec6832e5ca1767: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260929 17:55:39.401556 15179 raft_consensus.cc:348] T 34cb3888e81d468884bb1e9936a65cba P 3f14c4f2dc464bcb93ec6832e5ca1767 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f14c4f2dc464bcb93ec6832e5ca1767" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35183 } } peers { permanent_uuid: "a5ea02aa22e3423e947ffd890f843ee7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39167 } } peers { permanent_uuid: "40fc602f689f47a2afb4b5ddf158d470" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37811 } }
I20260929 17:55:39.401803 15179 raft_consensus.cc:374] T 34cb3888e81d468884bb1e9936a65cba P 3f14c4f2dc464bcb93ec6832e5ca1767 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:55:39.401841 15179 raft_consensus.cc:729] T 34cb3888e81d468884bb1e9936a65cba P 3f14c4f2dc464bcb93ec6832e5ca1767 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f14c4f2dc464bcb93ec6832e5ca1767, State: Initialized, Role: FOLLOWER
I20260929 17:55:39.402225 15179 consensus_queue.cc:260] T 34cb3888e81d468884bb1e9936a65cba P 3f14c4f2dc464bcb93ec6832e5ca1767 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f14c4f2dc464bcb93ec6832e5ca1767" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35183 } } peers { permanent_uuid: "a5ea02aa22e3423e947ffd890f843ee7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39167 } } peers { permanent_uuid: "40fc602f689f47a2afb4b5ddf158d470" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37811 } }
I20260929 17:55:39.402802 15030 heartbeater.cc:503] Master 127.0.98.62:44077 was elected leader, sending a full tablet report...
I20260929 17:55:39.403102 15179 ts_tablet_manager.cc:1428] T 34cb3888e81d468884bb1e9936a65cba P 3f14c4f2dc464bcb93ec6832e5ca1767: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
W20260929 17:55:39.459100 14893 tablet.cc:2369] T 34cb3888e81d468884bb1e9936a65cba P 40fc602f689f47a2afb4b5ddf158d470: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 17:55:39.584036 15165 tablet.cc:2369] T 34cb3888e81d468884bb1e9936a65cba P a5ea02aa22e3423e947ffd890f843ee7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 17:55:39.651001 15031 tablet.cc:2369] T 34cb3888e81d468884bb1e9936a65cba P 3f14c4f2dc464bcb93ec6832e5ca1767: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 17:55:39.660449 15181 raft_consensus.cc:482] T 34cb3888e81d468884bb1e9936a65cba P 3f14c4f2dc464bcb93ec6832e5ca1767 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 17:55:39.660646 15181 raft_consensus.cc:504] T 34cb3888e81d468884bb1e9936a65cba P 3f14c4f2dc464bcb93ec6832e5ca1767 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f14c4f2dc464bcb93ec6832e5ca1767" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35183 } } peers { permanent_uuid: "a5ea02aa22e3423e947ffd890f843ee7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39167 } } peers { permanent_uuid: "40fc602f689f47a2afb4b5ddf158d470" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37811 } }
I20260929 17:55:39.661890 15181 leader_election.cc:288] T 34cb3888e81d468884bb1e9936a65cba P 3f14c4f2dc464bcb93ec6832e5ca1767 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a5ea02aa22e3423e947ffd890f843ee7 (127.0.98.3:39167), 40fc602f689f47a2afb4b5ddf158d470 (127.0.98.1:37811)
I20260929 17:55:39.665679 15182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:55:39.662259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39331 (local address 127.0.98.1:37811)
0929 17:55:39.662547 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0929 17:55:39.662551 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:55:39.662565 (+ 14us) server_negotiation.cc:407] Connection header received
0929 17:55:39.662615 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:55:39.662617 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:55:39.662667 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:55:39.662761 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:55:39.663017 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:55:39.663612 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:55:39.664650 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:55:39.665145 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:55:39.665241 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:55:39.665411 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0929 17:55:39.665476 (+ 65us) server_negotiation.cc:299] Negotiation successful
0929 17:55:39.665546 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":218,"thread_start_us":99,"threads_started":1}
I20260929 17:55:39.665678 15033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:55:39.662129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37811 (local address 127.0.98.2:39331)
0929 17:55:39.662281 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0929 17:55:39.662297 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 17:55:39.662414 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:55:39.662784 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:55:39.662787 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:55:39.662805 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:55:39.662912 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:55:39.662917 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:55:39.663721 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:55:39.663724 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:55:39.664532 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:55:39.664541 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:55:39.665256 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:55:39.665260 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:55:39.665370 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:55:39.665392 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 17:55:39.665462 (+ 70us) client_negotiation.cc:240] Negotiation successful
0929 17:55:39.665540 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63,"mutex_wait_us":4}
I20260929 17:55:39.665771 15170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:55:39.662315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46519 (local address 127.0.98.3:39167)
0929 17:55:39.662443 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0929 17:55:39.662448 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:55:39.662714 (+ 266us) server_negotiation.cc:407] Connection header received
0929 17:55:39.662833 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:55:39.662836 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:55:39.662889 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:55:39.662988 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:55:39.663245 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:55:39.663788 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:55:39.664796 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:55:39.665251 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:55:39.665341 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:55:39.665499 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0929 17:55:39.665562 (+ 63us) server_negotiation.cc:299] Negotiation successful
0929 17:55:39.665629 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 17:55:39.665799 15183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:55:39.662129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39167 (local address 127.0.98.2:46519)
0929 17:55:39.662619 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0929 17:55:39.662629 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 17:55:39.662735 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:55:39.663004 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:55:39.663006 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:55:39.663021 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:55:39.663142 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:55:39.663148 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:55:39.663897 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:55:39.663901 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:55:39.664679 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:55:39.664687 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:55:39.665362 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:55:39.665365 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:55:39.665464 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:55:39.665485 (+ 21us) client_negotiation.cc:769] Sending connection context
0929 17:55:39.665552 (+ 67us) client_negotiation.cc:240] Negotiation successful
0929 17:55:39.665621 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":432,"mutex_wait_us":229,"thread_start_us":72,"threads_started":1}
I20260929 17:55:39.666412 15117 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34cb3888e81d468884bb1e9936a65cba" candidate_uuid: "3f14c4f2dc464bcb93ec6832e5ca1767" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5ea02aa22e3423e947ffd890f843ee7" is_pre_election: true
I20260929 17:55:39.666632 14846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34cb3888e81d468884bb1e9936a65cba" candidate_uuid: "3f14c4f2dc464bcb93ec6832e5ca1767" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40fc602f689f47a2afb4b5ddf158d470" is_pre_election: true
I20260929 17:55:39.666705 15117 raft_consensus.cc:2454] T 34cb3888e81d468884bb1e9936a65cba P a5ea02aa22e3423e947ffd890f843ee7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f14c4f2dc464bcb93ec6832e5ca1767 in term 0.
I20260929 17:55:39.666909 14846 raft_consensus.cc:2454] T 34cb3888e81d468884bb1e9936a65cba P 40fc602f689f47a2afb4b5ddf158d470 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f14c4f2dc464bcb93ec6832e5ca1767 in term 0.
I20260929 17:55:39.667145 14919 leader_election.cc:302] T 34cb3888e81d468884bb1e9936a65cba P 3f14c4f2dc464bcb93ec6832e5ca1767 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f14c4f2dc464bcb93ec6832e5ca1767, a5ea02aa22e3423e947ffd890f843ee7; no voters:
I20260929 17:55:39.667407 15181 raft_consensus.cc:2781] T 34cb3888e81d468884bb1e9936a65cba P 3f14c4f2dc464bcb93ec6832e5ca1767 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 17:55:39.667492 15181 raft_consensus.cc:482] T 34cb3888e81d468884bb1e9936a65cba P 3f14c4f2dc464bcb93ec6832e5ca1767 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 17:55:39.667536 15181 raft_consensus.cc:3037] T 34cb3888e81d468884bb1e9936a65cba P 3f14c4f2dc464bcb93ec6832e5ca1767 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:55:39.668521 15181 raft_consensus.cc:504] T 34cb3888e81d468884bb1e9936a65cba P 3f14c4f2dc464bcb93ec6832e5ca1767 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f14c4f2dc464bcb93ec6832e5ca1767" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35183 } } peers { permanent_uuid: "a5ea02aa22e3423e947ffd890f843ee7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39167 } } peers { permanent_uuid: "40fc602f689f47a2afb4b5ddf158d470" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37811 } }
I20260929 17:55:39.668859 15181 leader_election.cc:288] T 34cb3888e81d468884bb1e9936a65cba P 3f14c4f2dc464bcb93ec6832e5ca1767 [CANDIDATE]: Term 1 election: Requested vote from peers a5ea02aa22e3423e947ffd890f843ee7 (127.0.98.3:39167), 40fc602f689f47a2afb4b5ddf158d470 (127.0.98.1:37811)
I20260929 17:55:39.669220 14846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34cb3888e81d468884bb1e9936a65cba" candidate_uuid: "3f14c4f2dc464bcb93ec6832e5ca1767" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40fc602f689f47a2afb4b5ddf158d470"
I20260929 17:55:39.669220 15117 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34cb3888e81d468884bb1e9936a65cba" candidate_uuid: "3f14c4f2dc464bcb93ec6832e5ca1767" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5ea02aa22e3423e947ffd890f843ee7"
I20260929 17:55:39.669353 14846 raft_consensus.cc:3037] T 34cb3888e81d468884bb1e9936a65cba P 40fc602f689f47a2afb4b5ddf158d470 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:55:39.669353 15117 raft_consensus.cc:3037] T 34cb3888e81d468884bb1e9936a65cba P a5ea02aa22e3423e947ffd890f843ee7 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:55:39.670614 15117 raft_consensus.cc:2454] T 34cb3888e81d468884bb1e9936a65cba P a5ea02aa22e3423e947ffd890f843ee7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f14c4f2dc464bcb93ec6832e5ca1767 in term 1.
I20260929 17:55:39.670614 14846 raft_consensus.cc:2454] T 34cb3888e81d468884bb1e9936a65cba P 40fc602f689f47a2afb4b5ddf158d470 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f14c4f2dc464bcb93ec6832e5ca1767 in term 1.
I20260929 17:55:39.670902 14918 leader_election.cc:302] T 34cb3888e81d468884bb1e9936a65cba P 3f14c4f2dc464bcb93ec6832e5ca1767 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f14c4f2dc464bcb93ec6832e5ca1767, 40fc602f689f47a2afb4b5ddf158d470; no voters:
I20260929 17:55:39.671064 15181 raft_consensus.cc:2781] T 34cb3888e81d468884bb1e9936a65cba P 3f14c4f2dc464bcb93ec6832e5ca1767 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 17:55:39.671278 15181 raft_consensus.cc:686] T 34cb3888e81d468884bb1e9936a65cba P 3f14c4f2dc464bcb93ec6832e5ca1767 [term 1 LEADER]: Becoming Leader. State: Replica: 3f14c4f2dc464bcb93ec6832e5ca1767, State: Running, Role: LEADER
I20260929 17:55:39.671635 15181 consensus_queue.cc:237] T 34cb3888e81d468884bb1e9936a65cba P 3f14c4f2dc464bcb93ec6832e5ca1767 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f14c4f2dc464bcb93ec6832e5ca1767" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35183 } } peers { permanent_uuid: "a5ea02aa22e3423e947ffd890f843ee7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39167 } } peers { permanent_uuid: "40fc602f689f47a2afb4b5ddf158d470" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37811 } }
I20260929 17:55:39.674906 14704 catalog_manager.cc:5495] T 34cb3888e81d468884bb1e9936a65cba P 3f14c4f2dc464bcb93ec6832e5ca1767 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3f14c4f2dc464bcb93ec6832e5ca1767 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "3f14c4f2dc464bcb93ec6832e5ca1767" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f14c4f2dc464bcb93ec6832e5ca1767" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35183 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a5ea02aa22e3423e947ffd890f843ee7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39167 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "40fc602f689f47a2afb4b5ddf158d470" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37811 } health_report { overall_health: UNKNOWN } } }
I20260929 17:55:39.754246 15176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:55:39.750843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48216 (local address 127.0.98.2:35183)
0929 17:55:39.750980 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0929 17:55:39.750983 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 17:55:39.750998 (+ 15us) server_negotiation.cc:407] Connection header received
0929 17:55:39.751075 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:55:39.751077 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:55:39.751134 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:55:39.751220 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 17:55:39.751417 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:55:39.751972 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:55:39.752792 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:55:39.753048 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:55:39.753125 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:55:39.753372 (+ 247us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 17:55:39.753806 (+ 434us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 17:55:39.753896 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0929 17:55:39.754011 (+ 115us) server_negotiation.cc:299] Negotiation successful
0929 17:55:39.754066 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 17:55:39.760216 15117 raft_consensus.cc:1261] T 34cb3888e81d468884bb1e9936a65cba P a5ea02aa22e3423e947ffd890f843ee7 [term 1 FOLLOWER]: Refusing update from remote peer 3f14c4f2dc464bcb93ec6832e5ca1767: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 17:55:39.760217 14846 raft_consensus.cc:1261] T 34cb3888e81d468884bb1e9936a65cba P 40fc602f689f47a2afb4b5ddf158d470 [term 1 FOLLOWER]: Refusing update from remote peer 3f14c4f2dc464bcb93ec6832e5ca1767: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 17:55:39.760893 15181 consensus_queue.cc:1035] T 34cb3888e81d468884bb1e9936a65cba P 3f14c4f2dc464bcb93ec6832e5ca1767 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5ea02aa22e3423e947ffd890f843ee7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39167 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 17:55:39.761101 15184 consensus_queue.cc:1035] T 34cb3888e81d468884bb1e9936a65cba P 3f14c4f2dc464bcb93ec6832e5ca1767 [LEADER]: Connected to new peer: Peer: permanent_uuid: "40fc602f689f47a2afb4b5ddf158d470" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37811 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 17:55:39.767441 15187 mvcc.cc:205] Tried to move back new op lower bound from 7334725794850734080 to 7334725794497798144. Current Snapshot: MvccSnapshot[applied={T|T < 7334725794850734080}]
I20260929 17:55:39.778190 14704 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:45026:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 17:55:39.778339 14704 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:45026:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 17:55:39.780457 14704 catalog_manager.cc:5782] T 00000000000000000000000000000000 P de3ecf1eeb38458faf6a908987c915f1: Sending DeleteTablet for 3 replicas of tablet 34cb3888e81d468884bb1e9936a65cba
I20260929 17:55:39.781114 14964 tablet_service.cc:1514] Processing DeleteTablet for tablet 34cb3888e81d468884bb1e9936a65cba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 17:55:39 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48214
I20260929 17:55:39.781201 14826 tablet_service.cc:1514] Processing DeleteTablet for tablet 34cb3888e81d468884bb1e9936a65cba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 17:55:39 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51790
I20260929 17:55:39.781248 15097 tablet_service.cc:1514] Processing DeleteTablet for tablet 34cb3888e81d468884bb1e9936a65cba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 17:55:39 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45530
I20260929 17:55:39.781510 15172 tablet_replica.cc:330] T 34cb3888e81d468884bb1e9936a65cba P a5ea02aa22e3423e947ffd890f843ee7: stopping tablet replica
I20260929 17:55:39.781601 15172 raft_consensus.cc:2229] T 34cb3888e81d468884bb1e9936a65cba P a5ea02aa22e3423e947ffd890f843ee7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 17:55:39.781627 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskup3Soe/build/debug/bin/kudu with pid 14742
I20260929 17:55:39.781713 15172 raft_consensus.cc:2258] T 34cb3888e81d468884bb1e9936a65cba P a5ea02aa22e3423e947ffd890f843ee7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 17:55:39.782452 15197 tablet_replica.cc:330] T 34cb3888e81d468884bb1e9936a65cba P 3f14c4f2dc464bcb93ec6832e5ca1767: stopping tablet replica
I20260929 17:55:39.782630 15172 ts_tablet_manager.cc:1905] T 34cb3888e81d468884bb1e9936a65cba P a5ea02aa22e3423e947ffd890f843ee7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 17:55:39.782800 15197 raft_consensus.cc:2229] T 34cb3888e81d468884bb1e9936a65cba P 3f14c4f2dc464bcb93ec6832e5ca1767 [term 1 LEADER]: Raft consensus shutting down.
I20260929 17:55:39.783169 15197 raft_consensus.cc:2258] T 34cb3888e81d468884bb1e9936a65cba P 3f14c4f2dc464bcb93ec6832e5ca1767 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 17:55:39.784567 15197 ts_tablet_manager.cc:1905] T 34cb3888e81d468884bb1e9936a65cba P 3f14c4f2dc464bcb93ec6832e5ca1767: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 17:55:39.785219 15172 ts_tablet_manager.cc:1918] T 34cb3888e81d468884bb1e9936a65cba P a5ea02aa22e3423e947ffd890f843ee7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 17:55:39.785298 15172 log.cc:1196] T 34cb3888e81d468884bb1e9936a65cba P a5ea02aa22e3423e947ffd890f843ee7: Deleting WAL directory at /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/minicluster-data/ts-2/wal/wals/34cb3888e81d468884bb1e9936a65cba
I20260929 17:55:39.785544 15172 ts_tablet_manager.cc:1939] T 34cb3888e81d468884bb1e9936a65cba P a5ea02aa22e3423e947ffd890f843ee7: Deleting consensus metadata
I20260929 17:55:39.786315 14690 catalog_manager.cc:4841] TS a5ea02aa22e3423e947ffd890f843ee7 (127.0.98.3:39167): tablet 34cb3888e81d468884bb1e9936a65cba (table test-table [id=6bc0edaac07640729c421e8935be307b]) successfully deleted
I20260929 17:55:39.787246 15197 ts_tablet_manager.cc:1918] T 34cb3888e81d468884bb1e9936a65cba P 3f14c4f2dc464bcb93ec6832e5ca1767: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 17:55:39.787335 15197 log.cc:1196] T 34cb3888e81d468884bb1e9936a65cba P 3f14c4f2dc464bcb93ec6832e5ca1767: Deleting WAL directory at /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/minicluster-data/ts-1/wal/wals/34cb3888e81d468884bb1e9936a65cba
I20260929 17:55:39.787601 15197 ts_tablet_manager.cc:1939] T 34cb3888e81d468884bb1e9936a65cba P 3f14c4f2dc464bcb93ec6832e5ca1767: Deleting consensus metadata
I20260929 17:55:39.788664 14688 catalog_manager.cc:4841] TS 3f14c4f2dc464bcb93ec6832e5ca1767 (127.0.98.2:35183): tablet 34cb3888e81d468884bb1e9936a65cba (table test-table [id=6bc0edaac07640729c421e8935be307b]) successfully deleted
W20260929 17:55:39.790975 14689 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:37811 (error 108)
I20260929 17:55:39.791483 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskup3Soe/build/debug/bin/kudu with pid 14896
I20260929 17:55:39.792706 14899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:55:39.791750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37811 (local address 127.0.0.1:51798)
0929 17:55:39.792533 (+ 783us) negotiation.cc:107] Waiting for socket to connect
0929 17:55:39.792618 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":687}
W20260929 17:55:39.792928 14689 catalog_manager.cc:4584] TS 40fc602f689f47a2afb4b5ddf158d470 (127.0.98.1:37811): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 34cb3888e81d468884bb1e9936a65cba: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37811: connect: Connection refused (error 111)
I20260929 17:55:39.797173 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskup3Soe/build/debug/bin/kudu with pid 15032
I20260929 17:55:39.801892 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskup3Soe/build/debug/bin/kudu with pid 14671
I20260929 17:55:39.807674 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskup3Soe/build/debug/bin/kudu
/tmp/dist-test-taskup3Soe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44077
--webserver_interface=127.0.98.62
--webserver_port=37917
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44077
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:55:39.920558 15200 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:55:39.920878 15200 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:55:39.920950 15200 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:55:39.925212 15200 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 17:55:39.925290 15200 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:55:39.925318 15200 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:55:39.925339 15200 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 17:55:39.925360 15200 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 17:55:39.930867 15200 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44077
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44077
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=37917
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15200
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:55:39.931993 15200 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:55:39.933060 15200 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:55:39.941535 15205 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:55:39.941689 15200 server_base.cc:1034] running on GCE node
W20260929 17:55:39.941606 15208 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:55:39.941581 15206 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:55:39.942478 15200 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:55:39.943666 15200 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:55:39.944847 15200 hybrid_clock.cc:648] HybridClock initialized: now 1790704539944813 us; error 45 us; skew 500 ppm
Sep 29 17:55:39 dist-test-slave-53nh krb5kdc[14123](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704539, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:55:39.947577 15200 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 17:55:39.948424 15200 webserver.cc:460] Webserver started at http://127.0.98.62:37917/ using document root <none> and password file <none>
I20260929 17:55:39.948958 15200 fs_manager.cc:359] Metadata directory not provided
I20260929 17:55:39.949054 15200 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:55:39.952837 15200 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.002s
I20260929 17:55:39.954998 15215 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:55:39.956259 15200 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 17:55:39.956389 15200 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/minicluster-data/master-0/wal
uuid: "de3ecf1eeb38458faf6a908987c915f1"
format_stamp: "Formatted at 2026-09-29 17:55:37 on dist-test-slave-53nh"
server_key: "e54a357af30c51c020e49e8278319678"
server_key_iv: "447fc3407948338309cdb10f2b738ea7"
server_key_version: "encryptionkey@0"
I20260929 17:55:39.956945 15200 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:55:39.969143 15200 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:55:39.972330 15200 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:55:39.972507 15200 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:55:39.978829 15200 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44077
I20260929 17:55:39.978911 15267 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44077 every 8 connection(s)
I20260929 17:55:39.979847 15200 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/minicluster-data/master-0/data/info.pb
I20260929 17:55:39.983999 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskup3Soe/build/debug/bin/kudu as pid 15200
I20260929 17:55:39.984113 15268 sys_catalog.cc:263] Verifying existing consensus state
I20260929 17:55:39.984449 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskup3Soe/build/debug/bin/kudu
/tmp/dist-test-taskup3Soe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:37811
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:44077
--webserver_port=37677
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 17:55:39.986161 15268 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de3ecf1eeb38458faf6a908987c915f1: Bootstrap starting.
I20260929 17:55:40.002883 15268 log.cc:824] T 00000000000000000000000000000000 P de3ecf1eeb38458faf6a908987c915f1: Log is configured to *not* fsync() on all Append() calls
I20260929 17:55:40.014088 15268 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de3ecf1eeb38458faf6a908987c915f1: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 17:55:40.014446 15268 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de3ecf1eeb38458faf6a908987c915f1: Bootstrap complete.
I20260929 17:55:40.018456 15268 raft_consensus.cc:348] T 00000000000000000000000000000000 P de3ecf1eeb38458faf6a908987c915f1 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de3ecf1eeb38458faf6a908987c915f1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44077 } }
I20260929 17:55:40.018849 15268 raft_consensus.cc:729] T 00000000000000000000000000000000 P de3ecf1eeb38458faf6a908987c915f1 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: de3ecf1eeb38458faf6a908987c915f1, State: Initialized, Role: FOLLOWER
I20260929 17:55:40.019364 15268 consensus_queue.cc:260] T 00000000000000000000000000000000 P de3ecf1eeb38458faf6a908987c915f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de3ecf1eeb38458faf6a908987c915f1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44077 } }
I20260929 17:55:40.019512 15268 raft_consensus.cc:388] T 00000000000000000000000000000000 P de3ecf1eeb38458faf6a908987c915f1 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 17:55:40.019578 15268 raft_consensus.cc:482] T 00000000000000000000000000000000 P de3ecf1eeb38458faf6a908987c915f1 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 17:55:40.019663 15268 raft_consensus.cc:3037] T 00000000000000000000000000000000 P de3ecf1eeb38458faf6a908987c915f1 [term 2 FOLLOWER]: Advancing to term 3
I20260929 17:55:40.022279 15268 raft_consensus.cc:504] T 00000000000000000000000000000000 P de3ecf1eeb38458faf6a908987c915f1 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de3ecf1eeb38458faf6a908987c915f1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44077 } }
I20260929 17:55:40.022647 15268 leader_election.cc:302] T 00000000000000000000000000000000 P de3ecf1eeb38458faf6a908987c915f1 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: de3ecf1eeb38458faf6a908987c915f1; no voters:
I20260929 17:55:40.023021 15268 leader_election.cc:288] T 00000000000000000000000000000000 P de3ecf1eeb38458faf6a908987c915f1 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260929 17:55:40.023279 15272 raft_consensus.cc:2781] T 00000000000000000000000000000000 P de3ecf1eeb38458faf6a908987c915f1 [term 3 FOLLOWER]: Leader election won for term 3
I20260929 17:55:40.024199 15272 raft_consensus.cc:686] T 00000000000000000000000000000000 P de3ecf1eeb38458faf6a908987c915f1 [term 3 LEADER]: Becoming Leader. State: Replica: de3ecf1eeb38458faf6a908987c915f1, State: Running, Role: LEADER
I20260929 17:55:40.024369 15268 sys_catalog.cc:564] T 00000000000000000000000000000000 P de3ecf1eeb38458faf6a908987c915f1 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 17:55:40.025141 15272 consensus_queue.cc:237] T 00000000000000000000000000000000 P de3ecf1eeb38458faf6a908987c915f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de3ecf1eeb38458faf6a908987c915f1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44077 } }
I20260929 17:55:40.027066 15274 sys_catalog.cc:455] T 00000000000000000000000000000000 P de3ecf1eeb38458faf6a908987c915f1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader de3ecf1eeb38458faf6a908987c915f1. Latest consensus state: current_term: 3 leader_uuid: "de3ecf1eeb38458faf6a908987c915f1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de3ecf1eeb38458faf6a908987c915f1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44077 } } }
I20260929 17:55:40.027206 15274 sys_catalog.cc:458] T 00000000000000000000000000000000 P de3ecf1eeb38458faf6a908987c915f1 [sys.catalog]: This master's current role is: LEADER
I20260929 17:55:40.027374 15273 sys_catalog.cc:455] T 00000000000000000000000000000000 P de3ecf1eeb38458faf6a908987c915f1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "de3ecf1eeb38458faf6a908987c915f1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de3ecf1eeb38458faf6a908987c915f1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44077 } } }
I20260929 17:55:40.027869 15273 sys_catalog.cc:458] T 00000000000000000000000000000000 P de3ecf1eeb38458faf6a908987c915f1 [sys.catalog]: This master's current role is: LEADER
I20260929 17:55:40.027781 15287 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 17:55:40.032743 15287 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 17:55:40.033385 15287 catalog_manager.cc:1259] Loaded cluster ID: 374f9335722247a4aa9886591ecde9e8
I20260929 17:55:40.033453 15287 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260929 17:55:40.034368 15287 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f60423a29ad google::LogMessage::Fail() at ??:0
@ 0x7f60423a588a google::LogMessage::SendToLog() at ??:0
@ 0x7f60423a24de google::LogMessage::Flush() at ??:0
@ 0x7f60423a3a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f604cce1fdd _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f604cce2d2f kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f604ccdf579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f604cd23640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f60445f43d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f60445f4c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f60445f63f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f60445e5512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f6045c296db start_thread at ??:0
@ 0x7f6040ea061f clone at ??:0
W20260929 17:55:40.110906 15270 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:55:40.111148 15270 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:55:40.111191 15270 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:55:40.115528 15270 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:55:40.115579 15270 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:55:40.115639 15270 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 17:55:40.121160 15270 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:37811
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=37677
--tserver_master_addrs=127.0.98.62:44077
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15270
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:55:40.122246 15270 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:55:40.123260 15270 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:55:40.132823 15309 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:55:40.132860 15312 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:55:40.133086 15270 server_base.cc:1034] running on GCE node
W20260929 17:55:40.133208 15310 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:55:40.133638 15270 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:55:40.134279 15270 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:55:40.135422 15270 hybrid_clock.cc:648] HybridClock initialized: now 1790704540135388 us; error 54 us; skew 500 ppm
Sep 29 17:55:40 dist-test-slave-53nh krb5kdc[14123](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704540, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:55:40.138104 15270 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260929 17:55:40.138969 15270 webserver.cc:460] Webserver started at http://127.0.98.1:37677/ using document root <none> and password file <none>
I20260929 17:55:40.139499 15270 fs_manager.cc:359] Metadata directory not provided
I20260929 17:55:40.139557 15270 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:55:40.143661 15270 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260929 17:55:40.146092 15319 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:55:40.147622 15270 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 17:55:40.147763 15270 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/minicluster-data/ts-0/wal
uuid: "40fc602f689f47a2afb4b5ddf158d470"
format_stamp: "Formatted at 2026-09-29 17:55:37 on dist-test-slave-53nh"
server_key: "33beeb6c59ac0ce99be2d097572343fc"
server_key_iv: "fdb9495491df392eba099ab69555576f"
server_key_version: "encryptionkey@0"
I20260929 17:55:40.148367 15270 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:55:40.173270 15270 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:55:40.177114 15270 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:55:40.177320 15270 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:55:40.178099 15270 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 17:55:40.179453 15326 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260929 17:55:40.183545 15270 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260929 17:55:40.183629 15270 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.001s sys 0.000s
I20260929 17:55:40.183709 15270 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260929 17:55:40.185945 15270 ts_tablet_manager.cc:610] Registered 1 tablets
I20260929 17:55:40.186005 15270 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.002s sys 0.000s
I20260929 17:55:40.186271 15326 tablet_bootstrap.cc:492] T 34cb3888e81d468884bb1e9936a65cba P 40fc602f689f47a2afb4b5ddf158d470: Bootstrap starting.
I20260929 17:55:40.196704 15270 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37811
I20260929 17:55:40.196743 15433 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37811 every 8 connection(s)
I20260929 17:55:40.197926 15270 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/minicluster-data/ts-0/data/info.pb
I20260929 17:55:40.201572 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskup3Soe/build/debug/bin/kudu as pid 15270
I20260929 17:55:40.202049 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskup3Soe/build/debug/bin/kudu
/tmp/dist-test-taskup3Soe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:35183
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:44077
--webserver_port=43661
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 17:55:40.205394 15326 log.cc:824] T 34cb3888e81d468884bb1e9936a65cba P 40fc602f689f47a2afb4b5ddf158d470: Log is configured to *not* fsync() on all Append() calls
I20260929 17:55:40.214116 15326 tablet_bootstrap.cc:492] T 34cb3888e81d468884bb1e9936a65cba P 40fc602f689f47a2afb4b5ddf158d470: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 17:55:40.214530 15326 tablet_bootstrap.cc:492] T 34cb3888e81d468884bb1e9936a65cba P 40fc602f689f47a2afb4b5ddf158d470: Bootstrap complete.
I20260929 17:55:40.215832 15326 ts_tablet_manager.cc:1397] T 34cb3888e81d468884bb1e9936a65cba P 40fc602f689f47a2afb4b5ddf158d470: Time spent bootstrapping tablet: real 0.030s user 0.022s sys 0.004s
I20260929 17:55:40.219000 15326 raft_consensus.cc:348] T 34cb3888e81d468884bb1e9936a65cba P 40fc602f689f47a2afb4b5ddf158d470 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f14c4f2dc464bcb93ec6832e5ca1767" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35183 } } peers { permanent_uuid: "a5ea02aa22e3423e947ffd890f843ee7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39167 } } peers { permanent_uuid: "40fc602f689f47a2afb4b5ddf158d470" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37811 } }
I20260929 17:55:40.219429 15326 raft_consensus.cc:729] T 34cb3888e81d468884bb1e9936a65cba P 40fc602f689f47a2afb4b5ddf158d470 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40fc602f689f47a2afb4b5ddf158d470, State: Initialized, Role: FOLLOWER
I20260929 17:55:40.219959 15326 consensus_queue.cc:260] T 34cb3888e81d468884bb1e9936a65cba P 40fc602f689f47a2afb4b5ddf158d470 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f14c4f2dc464bcb93ec6832e5ca1767" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35183 } } peers { permanent_uuid: "a5ea02aa22e3423e947ffd890f843ee7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39167 } } peers { permanent_uuid: "40fc602f689f47a2afb4b5ddf158d470" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37811 } }
I20260929 17:55:40.221053 15326 ts_tablet_manager.cc:1428] T 34cb3888e81d468884bb1e9936a65cba P 40fc602f689f47a2afb4b5ddf158d470: Time spent starting tablet: real 0.005s user 0.008s sys 0.000s
W20260929 17:55:40.269053 15436 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:55:40.200272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44077 (local address 127.0.98.1:43345)
0929 17:55:40.200869 (+ 597us) negotiation.cc:107] Waiting for socket to connect
0929 17:55:40.200929 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0929 17:55:40.202390 (+ 1461us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:55:40.268016 (+ 65626us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44077: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":161,"threads_started":1}
W20260929 17:55:40.269687 15434 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:44077 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:44077: Client connection negotiation failed: client connection to 127.0.98.62:44077: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260929 17:55:40.270464 15436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:55:40.270163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44077 (local address 127.0.98.1:59523)
0929 17:55:40.270310 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0929 17:55:40.270386 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260929 17:55:40.271113 15436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:55:40.270898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44077 (local address 127.0.98.1:36019)
0929 17:55:40.271005 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0929 17:55:40.271054 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260929 17:55:40.271296 15434 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 17:55:40.313657 15437 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:55:40.313966 15437 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:55:40.314033 15437 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:55:40.318261 15437 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:55:40.318333 15437 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:55:40.318420 15437 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 17:55:40.323897 15437 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:35183
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=43661
--tserver_master_addrs=127.0.98.62:44077
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15437
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:55:40.324982 15437 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:55:40.326056 15437 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:55:40.335678 15447 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:55:40.335721 15437 server_base.cc:1034] running on GCE node
W20260929 17:55:40.335812 15444 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:55:40.335999 15445 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:55:40.336422 15437 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:55:40.337114 15437 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:55:40.338297 15437 hybrid_clock.cc:648] HybridClock initialized: now 1790704540338274 us; error 33 us; skew 500 ppm
Sep 29 17:55:40 dist-test-slave-53nh krb5kdc[14123](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704540, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:55:40.340979 15437 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260929 17:55:40.341876 15437 webserver.cc:460] Webserver started at http://127.0.98.2:43661/ using document root <none> and password file <none>
I20260929 17:55:40.342391 15437 fs_manager.cc:359] Metadata directory not provided
I20260929 17:55:40.342450 15437 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:55:40.346475 15437 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 17:55:40.348471 15454 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:55:40.349761 15437 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 17:55:40.349880 15437 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/minicluster-data/ts-1/wal
uuid: "3f14c4f2dc464bcb93ec6832e5ca1767"
format_stamp: "Formatted at 2026-09-29 17:55:37 on dist-test-slave-53nh"
server_key: "8621cc99ae06ddd2899c6618480993fb"
server_key_iv: "cf2b8eef85248132087c75353cfef9b9"
server_key_version: "encryptionkey@0"
I20260929 17:55:40.350421 15437 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:55:40.380483 15437 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:55:40.383571 15437 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:55:40.383725 15437 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:55:40.384291 15437 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 17:55:40.385126 15437 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:55:40.385174 15437 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:55:40.385214 15437 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:55:40.385233 15437 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 17:55:40.394057 15437 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35183
I20260929 17:55:40.394150 15567 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35183 every 8 connection(s)
I20260929 17:55:40.395129 15437 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/minicluster-data/ts-1/data/info.pb
I20260929 17:55:40.399139 15570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:55:40.397356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44077 (local address 127.0.98.2:40309)
0929 17:55:40.397828 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0929 17:55:40.397942 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":111,"threads_started":1}
I20260929 17:55:40.399539 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskup3Soe/build/debug/bin/kudu as pid 15437
W20260929 17:55:40.399606 15568 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:44077 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:44077: Client connection negotiation failed: client connection to 127.0.98.62:44077: connect: Connection refused (error 111)
I20260929 17:55:40.400020 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskup3Soe/build/debug/bin/kudu
/tmp/dist-test-taskup3Soe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:39167
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:44077
--webserver_port=45201
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 17:55:40.400273 15570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:55:40.400008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44077 (local address 127.0.98.2:50967)
0929 17:55:40.400152 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0929 17:55:40.400216 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260929 17:55:40.400851 15570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:55:40.400678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44077 (local address 127.0.98.2:42207)
0929 17:55:40.400764 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0929 17:55:40.400802 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260929 17:55:40.401048 15568 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 17:55:40.506753 15571 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:55:40.507016 15571 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:55:40.507073 15571 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:55:40.511019 15571 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:55:40.511080 15571 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:55:40.511158 15571 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 17:55:40.516253 15571 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:39167
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=45201
--tserver_master_addrs=127.0.98.62:44077
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15571
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:55:40.517323 15571 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:55:40.518306 15571 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:55:40.527498 15576 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:55:40.527562 15577 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:55:40.527504 15579 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:55:40.527673 15571 server_base.cc:1034] running on GCE node
I20260929 17:55:40.528146 15571 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:55:40.528751 15571 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:55:40.529922 15571 hybrid_clock.cc:648] HybridClock initialized: now 1790704540529895 us; error 38 us; skew 500 ppm
Sep 29 17:55:40 dist-test-slave-53nh krb5kdc[14123](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704540, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:55:40.532398 15571 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260929 17:55:40.533231 15571 webserver.cc:460] Webserver started at http://127.0.98.3:45201/ using document root <none> and password file <none>
I20260929 17:55:40.533715 15571 fs_manager.cc:359] Metadata directory not provided
I20260929 17:55:40.533769 15571 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:55:40.537428 15571 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260929 17:55:40.539321 15586 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:55:40.540467 15571 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 17:55:40.540556 15571 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/minicluster-data/ts-2/wal
uuid: "a5ea02aa22e3423e947ffd890f843ee7"
format_stamp: "Formatted at 2026-09-29 17:55:38 on dist-test-slave-53nh"
server_key: "d7f9e48bc909ed527929a4329e9d2231"
server_key_iv: "13942ed9259de28f8779a6c5d76a60ba"
server_key_version: "encryptionkey@0"
I20260929 17:55:40.541112 15571 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:55:40.549618 15571 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:55:40.552685 15571 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:55:40.552860 15571 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:55:40.553498 15571 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 17:55:40.554332 15571 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:55:40.554387 15571 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:55:40.554450 15571 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:55:40.554479 15571 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 17:55:40.562844 15571 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39167
I20260929 17:55:40.562912 15699 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39167 every 8 connection(s)
I20260929 17:55:40.563839 15571 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790704375975685-392-0/minicluster-data/ts-2/data/info.pb
I20260929 17:55:40.565790 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskup3Soe/build/debug/bin/kudu as pid 15571
I20260929 17:55:40.567849 15702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:55:40.565961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44077 (local address 127.0.98.3:56393)
0929 17:55:40.566447 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0929 17:55:40.567117 (+ 670us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":133,"threads_started":1}
W20260929 17:55:40.568323 15700 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:44077 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:44077: Client connection negotiation failed: client connection to 127.0.98.62:44077: connect: Connection refused (error 111)
I20260929 17:55:40.568979 15702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:55:40.568686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44077 (local address 127.0.98.3:39253)
0929 17:55:40.568857 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0929 17:55:40.568918 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
I20260929 17:55:40.569587 15702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:55:40.569408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44077 (local address 127.0.98.3:47949)
0929 17:55:40.569498 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0929 17:55:40.569539 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260929 17:55:40.569742 15700 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260929 17:55:41.272429 15704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:55:41.271957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44077 (local address 127.0.98.1:42373)
0929 17:55:41.272204 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0929 17:55:41.272293 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":80,"threads_started":1}
I20260929 17:55:41.402011 15705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:55:41.401582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44077 (local address 127.0.98.2:45065)
0929 17:55:41.401846 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0929 17:55:41.401901 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":71,"threads_started":1}
I20260929 17:55:41.522815 15706 raft_consensus.cc:482] T 34cb3888e81d468884bb1e9936a65cba P 40fc602f689f47a2afb4b5ddf158d470 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 17:55:41.523005 15706 raft_consensus.cc:504] T 34cb3888e81d468884bb1e9936a65cba P 40fc602f689f47a2afb4b5ddf158d470 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f14c4f2dc464bcb93ec6832e5ca1767" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35183 } } peers { permanent_uuid: "a5ea02aa22e3423e947ffd890f843ee7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39167 } } peers { permanent_uuid: "40fc602f689f47a2afb4b5ddf158d470" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37811 } }
I20260929 17:55:41.524048 15706 leader_election.cc:288] T 34cb3888e81d468884bb1e9936a65cba P 40fc602f689f47a2afb4b5ddf158d470 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3f14c4f2dc464bcb93ec6832e5ca1767 (127.0.98.2:35183), a5ea02aa22e3423e947ffd890f843ee7 (127.0.98.3:39167)
Sep 29 17:55:41 dist-test-slave-53nh krb5kdc[14123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704540, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
Sep 29 17:55:41 dist-test-slave-53nh krb5kdc[14123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704540, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
I20260929 17:55:41.536917 15708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:55:41.524422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53697 (local address 127.0.98.2:35183)
0929 17:55:41.524798 (+ 376us) server_negotiation.cc:206] Beginning negotiation
0929 17:55:41.524815 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 17:55:41.524878 (+ 63us) server_negotiation.cc:407] Connection header received
0929 17:55:41.525606 (+ 728us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:55:41.525613 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:55:41.525903 (+ 290us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:55:41.526178 (+ 275us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:55:41.527409 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:55:41.528338 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:55:41.529536 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:55:41.529980 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:55:41.530119 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:55:41.533098 (+ 2979us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:55:41.533106 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:55:41.533118 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:55:41.533145 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:55:41.534724 (+ 1579us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:55:41.535118 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:55:41.535122 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:55:41.535126 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:55:41.535229 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:55:41.535523 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:55:41.535526 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:55:41.535528 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:55:41.535902 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:55:41.536055 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0929 17:55:41.536523 (+ 468us) server_negotiation.cc:299] Negotiation successful
0929 17:55:41.536759 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":86,"threads_started":1}
I20260929 17:55:41.536908 15704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:55:41.524276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39167 (local address 127.0.98.1:51945)
0929 17:55:41.524367 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0929 17:55:41.524382 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 17:55:41.524496 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:55:41.526253 (+ 1757us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:55:41.526259 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:55:41.526786 (+ 527us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:55:41.527253 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:55:41.527267 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:55:41.528473 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:55:41.528476 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:55:41.529436 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:55:41.529445 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:55:41.530091 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:55:41.530095 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:55:41.530281 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:55:41.530884 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:55:41.530899 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:55:41.532534 (+ 1635us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:55:41.534643 (+ 2109us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:55:41.534649 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:55:41.534663 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:55:41.534979 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:55:41.535295 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:55:41.535299 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:55:41.535300 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:55:41.535448 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:55:41.536061 (+ 613us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:55:41.536064 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:55:41.536241 (+ 177us) client_negotiation.cc:769] Sending connection context
0929 17:55:41.536397 (+ 156us) client_negotiation.cc:240] Negotiation successful
0929 17:55:41.536717 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23,"mutex_wait_us":80,"spinlock_wait_cycles":9728}
I20260929 17:55:41.536942 15709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:55:41.524440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51945 (local address 127.0.98.3:39167)
0929 17:55:41.524741 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0929 17:55:41.524759 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 17:55:41.524803 (+ 44us) server_negotiation.cc:407] Connection header received
0929 17:55:41.525539 (+ 736us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:55:41.525545 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:55:41.525903 (+ 358us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:55:41.526177 (+ 274us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:55:41.527411 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:55:41.528356 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:55:41.529546 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:55:41.529979 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:55:41.530119 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:55:41.532759 (+ 2640us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:55:41.532764 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:55:41.532776 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:55:41.532802 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:55:41.534538 (+ 1736us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:55:41.535075 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:55:41.535081 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:55:41.535087 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:55:41.535210 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:55:41.535534 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:55:41.535537 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:55:41.535539 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:55:41.535952 (+ 413us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:55:41.536073 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0929 17:55:41.536518 (+ 445us) server_negotiation.cc:299] Negotiation successful
0929 17:55:41.536762 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":183,"thread_start_us":72,"threads_started":1}
I20260929 17:55:41.536952 15707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:55:41.524276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35183 (local address 127.0.98.1:53697)
0929 17:55:41.524712 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0929 17:55:41.524724 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 17:55:41.524811 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:55:41.526253 (+ 1442us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:55:41.526259 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:55:41.526786 (+ 527us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:55:41.527253 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:55:41.527267 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:55:41.528469 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:55:41.528473 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:55:41.529424 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:55:41.529433 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:55:41.530091 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:55:41.530095 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:55:41.530286 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:55:41.530873 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:55:41.530910 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:55:41.532845 (+ 1935us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:55:41.534825 (+ 1980us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:55:41.534828 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:55:41.534829 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:55:41.535025 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:55:41.535315 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:55:41.535318 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:55:41.535319 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:55:41.535438 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:55:41.536018 (+ 580us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:55:41.536025 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:55:41.536242 (+ 217us) client_negotiation.cc:769] Sending connection context
0929 17:55:41.536398 (+ 156us) client_negotiation.cc:240] Negotiation successful
0929 17:55:41.536787 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":376,"mutex_wait_us":448,"spinlock_wait_cycles":53504,"thread_start_us":70,"threads_started":1}
W20260929 17:55:41.539387 15320 leader_election.cc:341] T 34cb3888e81d468884bb1e9936a65cba P 40fc602f689f47a2afb4b5ddf158d470 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3f14c4f2dc464bcb93ec6832e5ca1767 (127.0.98.2:35183): Not found: Tablet not found: 34cb3888e81d468884bb1e9936a65cba
W20260929 17:55:41.539538 15322 leader_election.cc:341] T 34cb3888e81d468884bb1e9936a65cba P 40fc602f689f47a2afb4b5ddf158d470 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a5ea02aa22e3423e947ffd890f843ee7 (127.0.98.3:39167): Not found: Tablet not found: 34cb3888e81d468884bb1e9936a65cba
I20260929 17:55:41.539619 15322 leader_election.cc:302] T 34cb3888e81d468884bb1e9936a65cba P 40fc602f689f47a2afb4b5ddf158d470 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 40fc602f689f47a2afb4b5ddf158d470; no voters: 3f14c4f2dc464bcb93ec6832e5ca1767, a5ea02aa22e3423e947ffd890f843ee7
I20260929 17:55:41.539849 15706 raft_consensus.cc:2726] T 34cb3888e81d468884bb1e9936a65cba P 40fc602f689f47a2afb4b5ddf158d470 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 17:55:41.570660 15710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:55:41.570267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44077 (local address 127.0.98.3:38313)
0929 17:55:41.570509 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0929 17:55:41.570563 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":85,"threads_started":1}
I20260929 17:55:42.273731 15711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:55:42.273286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44077 (local address 127.0.98.1:51051)
0929 17:55:42.273568 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0929 17:55:42.273636 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":78,"threads_started":1}
I20260929 17:55:42.403102 15712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:55:42.402711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44077 (local address 127.0.98.2:41285)
0929 17:55:42.402952 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0929 17:55:42.403013 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":71,"threads_started":1}
I20260929 17:55:42.571820 15713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:55:42.571380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44077 (local address 127.0.98.3:58837)
0929 17:55:42.571639 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0929 17:55:42.571703 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":77,"threads_started":1}
I20260929 17:55:43.274996 15714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:55:43.274535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44077 (local address 127.0.98.1:60807)
0929 17:55:43.274829 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0929 17:55:43.274897 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":75,"threads_started":1}
I20260929 17:55:43.404338 15715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:55:43.403878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44077 (local address 127.0.98.2:53741)
0929 17:55:43.404161 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0929 17:55:43.404226 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":76,"threads_started":1}
I20260929 17:55:43.572916 15716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:55:43.572514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44077 (local address 127.0.98.3:44723)
0929 17:55:43.572758 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0929 17:55:43.572814 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":85,"threads_started":1}
I20260929 17:55:43.579193 15717 raft_consensus.cc:482] T 34cb3888e81d468884bb1e9936a65cba P 40fc602f689f47a2afb4b5ddf158d470 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 17:55:43.579313 15717 raft_consensus.cc:504] T 34cb3888e81d468884bb1e9936a65cba P 40fc602f689f47a2afb4b5ddf158d470 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f14c4f2dc464bcb93ec6832e5ca1767" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35183 } } peers { permanent_uuid: "a5ea02aa22e3423e947ffd890f843ee7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39167 } } peers { permanent_uuid: "40fc602f689f47a2afb4b5ddf158d470" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37811 } }
I20260929 17:55:43.579702 15717 leader_election.cc:288] T 34cb3888e81d468884bb1e9936a65cba P 40fc602f689f47a2afb4b5ddf158d470 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3f14c4f2dc464bcb93ec6832e5ca1767 (127.0.98.2:35183), a5ea02aa22e3423e947ffd890f843ee7 (127.0.98.3:39167)
W20260929 17:55:43.580484 15320 leader_election.cc:341] T 34cb3888e81d468884bb1e9936a65cba P 40fc602f689f47a2afb4b5ddf158d470 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3f14c4f2dc464bcb93ec6832e5ca1767 (127.0.98.2:35183): Not found: Tablet not found: 34cb3888e81d468884bb1e9936a65cba
W20260929 17:55:43.580593 15322 leader_election.cc:341] T 34cb3888e81d468884bb1e9936a65cba P 40fc602f689f47a2afb4b5ddf158d470 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a5ea02aa22e3423e947ffd890f843ee7 (127.0.98.3:39167): Not found: Tablet not found: 34cb3888e81d468884bb1e9936a65cba
I20260929 17:55:43.580680 15322 leader_election.cc:302] T 34cb3888e81d468884bb1e9936a65cba P 40fc602f689f47a2afb4b5ddf158d470 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 40fc602f689f47a2afb4b5ddf158d470; no voters: 3f14c4f2dc464bcb93ec6832e5ca1767, a5ea02aa22e3423e947ffd890f843ee7
I20260929 17:55:43.580821 15717 raft_consensus.cc:2726] T 34cb3888e81d468884bb1e9936a65cba P 40fc602f689f47a2afb4b5ddf158d470 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 17:55:44.276229 15718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:55:44.275805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44077 (local address 127.0.98.1:32899)
0929 17:55:44.276060 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0929 17:55:44.276123 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":83,"threads_started":1}
I20260929 17:55:44.405550 15719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:55:44.405163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44077 (local address 127.0.98.2:57005)
0929 17:55:44.405399 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0929 17:55:44.405461 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":73,"threads_started":1}
I20260929 17:55:44.574013 15720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:55:44.573599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44077 (local address 127.0.98.3:46925)
0929 17:55:44.573850 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0929 17:55:44.573909 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":69,"threads_started":1}
I20260929 17:55:45.277788 15721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:55:45.277275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44077 (local address 127.0.98.1:36367)
0929 17:55:45.277616 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0929 17:55:45.277685 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":78,"threads_started":1}
I20260929 17:55:45.406752 15722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:55:45.406314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44077 (local address 127.0.98.2:49341)
0929 17:55:45.406584 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0929 17:55:45.406646 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":76,"threads_started":1}
I20260929 17:55:45.575177 15723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:55:45.574735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44077 (local address 127.0.98.3:35465)
0929 17:55:45.575024 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0929 17:55:45.575081 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":106,"threads_started":1}
I20260929 17:55:45.730180 15724 raft_consensus.cc:482] T 34cb3888e81d468884bb1e9936a65cba P 40fc602f689f47a2afb4b5ddf158d470 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 17:55:45.730351 15724 raft_consensus.cc:504] T 34cb3888e81d468884bb1e9936a65cba P 40fc602f689f47a2afb4b5ddf158d470 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f14c4f2dc464bcb93ec6832e5ca1767" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35183 } } peers { permanent_uuid: "a5ea02aa22e3423e947ffd890f843ee7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39167 } } peers { permanent_uuid: "40fc602f689f47a2afb4b5ddf158d470" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37811 } }
I20260929 17:55:45.730760 15724 leader_election.cc:288] T 34cb3888e81d468884bb1e9936a65cba P 40fc602f689f47a2afb4b5ddf158d470 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3f14c4f2dc464bcb93ec6832e5ca1767 (127.0.98.2:35183), a5ea02aa22e3423e947ffd890f843ee7 (127.0.98.3:39167)
W20260929 17:55:45.731565 15322 leader_election.cc:341] T 34cb3888e81d468884bb1e9936a65cba P 40fc602f689f47a2afb4b5ddf158d470 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a5ea02aa22e3423e947ffd890f843ee7 (127.0.98.3:39167): Not found: Tablet not found: 34cb3888e81d468884bb1e9936a65cba
W20260929 17:55:45.731722 15320 leader_election.cc:341] T 34cb3888e81d468884bb1e9936a65cba P 40fc602f689f47a2afb4b5ddf158d470 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3f14c4f2dc464bcb93ec6832e5ca1767 (127.0.98.2:35183): Not found: Tablet not found: 34cb3888e81d468884bb1e9936a65cba
I20260929 17:55:45.731832 15320 leader_election.cc:302] T 34cb3888e81d468884bb1e9936a65cba P 40fc602f689f47a2afb4b5ddf158d470 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 40fc602f689f47a2afb4b5ddf158d470; no voters: 3f14c4f2dc464bcb93ec6832e5ca1767, a5ea02aa22e3423e947ffd890f843ee7
I20260929 17:55:45.731978 15724 raft_consensus.cc:2726] T 34cb3888e81d468884bb1e9936a65cba P 40fc602f689f47a2afb4b5ddf158d470 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 17:55:46.278999 15725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:55:46.278497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44077 (local address 127.0.98.1:53537)
0929 17:55:46.278822 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0929 17:55:46.278891 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":77,"threads_started":1}
I20260929 17:55:46.407974 15726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:55:46.407568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44077 (local address 127.0.98.2:60535)
0929 17:55:46.407810 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0929 17:55:46.407873 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":76,"threads_started":1}
I20260929 17:55:46.576428 15727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:55:46.575983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44077 (local address 127.0.98.3:59163)
0929 17:55:46.576255 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0929 17:55:46.576315 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":88,"threads_started":1}
I20260929 17:55:47.280162 15728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:55:47.279726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44077 (local address 127.0.98.1:34417)
0929 17:55:47.280007 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0929 17:55:47.280068 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":77,"threads_started":1}
I20260929 17:55:47.409230 15729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:55:47.408694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44077 (local address 127.0.98.2:49209)
0929 17:55:47.409075 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0929 17:55:47.409136 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":98,"threads_started":1}
I20260929 17:55:47.577672 15730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:55:47.577201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44077 (local address 127.0.98.3:43005)
0929 17:55:47.577512 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0929 17:55:47.577576 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":90,"threads_started":1}
I20260929 17:55:48.281397 15731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:55:48.280916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44077 (local address 127.0.98.1:58891)
0929 17:55:48.281234 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0929 17:55:48.281299 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":122,"threads_started":1}
I20260929 17:55:48.410490 15732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:55:48.410044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44077 (local address 127.0.98.2:52695)
0929 17:55:48.410320 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0929 17:55:48.410390 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":75,"threads_started":1}
I20260929 17:55:48.487028 15733 raft_consensus.cc:482] T 34cb3888e81d468884bb1e9936a65cba P 40fc602f689f47a2afb4b5ddf158d470 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 17:55:48.487166 15733 raft_consensus.cc:504] T 34cb3888e81d468884bb1e9936a65cba P 40fc602f689f47a2afb4b5ddf158d470 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f14c4f2dc464bcb93ec6832e5ca1767" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35183 } } peers { permanent_uuid: "a5ea02aa22e3423e947ffd890f843ee7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39167 } } peers { permanent_uuid: "40fc602f689f47a2afb4b5ddf158d470" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37811 } }
I20260929 17:55:48.487596 15733 leader_election.cc:288] T 34cb3888e81d468884bb1e9936a65cba P 40fc602f689f47a2afb4b5ddf158d470 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3f14c4f2dc464bcb93ec6832e5ca1767 (127.0.98.2:35183), a5ea02aa22e3423e947ffd890f843ee7 (127.0.98.3:39167)
W20260929 17:55:48.488318 15322 leader_election.cc:341] T 34cb3888e81d468884bb1e9936a65cba P 40fc602f689f47a2afb4b5ddf158d470 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a5ea02aa22e3423e947ffd890f843ee7 (127.0.98.3:39167): Not found: Tablet not found: 34cb3888e81d468884bb1e9936a65cba
W20260929 17:55:48.488459 15320 leader_election.cc:341] T 34cb3888e81d468884bb1e9936a65cba P 40fc602f689f47a2afb4b5ddf158d470 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3f14c4f2dc464bcb93ec6832e5ca1767 (127.0.98.2:35183): Not found: Tablet not found: 34cb3888e81d468884bb1e9936a65cba
I20260929 17:55:48.488539 15320 leader_election.cc:302] T 34cb3888e81d468884bb1e9936a65cba P 40fc602f689f47a2afb4b5ddf158d470 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 40fc602f689f47a2afb4b5ddf158d470; no voters: 3f14c4f2dc464bcb93ec6832e5ca1767, a5ea02aa22e3423e947ffd890f843ee7
I20260929 17:55:48.488682 15733 raft_consensus.cc:2726] T 34cb3888e81d468884bb1e9936a65cba P 40fc602f689f47a2afb4b5ddf158d470 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 17:55:48.578697 15734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:55:48.578319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44077 (local address 127.0.98.3:40379)
0929 17:55:48.578550 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0929 17:55:48.578606 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":78,"threads_started":1}
I20260929 17:55:49.282575 15735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:55:49.282150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44077 (local address 127.0.98.1:44553)
0929 17:55:49.282412 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0929 17:55:49.282477 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":92,"threads_started":1}
I20260929 17:55:49.411748 15736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:55:49.411307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44077 (local address 127.0.98.2:58411)
0929 17:55:49.411596 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0929 17:55:49.411655 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":79,"threads_started":1}
I20260929 17:55:49.579792 15737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:55:49.579383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44077 (local address 127.0.98.3:39255)
0929 17:55:49.579642 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0929 17:55:49.579697 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":76,"threads_started":1}
I20260929 17:55:50.283883 15738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:55:50.283456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44077 (local address 127.0.98.1:51949)
0929 17:55:50.283693 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0929 17:55:50.283755 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":76,"threads_started":1}
I20260929 17:55:50.413082 15739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:55:50.412627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44077 (local address 127.0.98.2:60853)
0929 17:55:50.412874 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0929 17:55:50.412945 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":76,"threads_started":1}
I20260929 17:55:50.580961 15740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:55:50.580523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44077 (local address 127.0.98.3:35611)
0929 17:55:50.580814 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0929 17:55:50.580867 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":74,"threads_started":1}
I20260929 17:55:51.285094 15741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:55:51.284637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44077 (local address 127.0.98.1:41895)
0929 17:55:51.284901 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0929 17:55:51.284960 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":72,"threads_started":1}
I20260929 17:55:51.414256 15742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:55:51.413835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44077 (local address 127.0.98.2:57865)
0929 17:55:51.414099 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0929 17:55:51.414161 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":78,"threads_started":1}
I20260929 17:55:51.582175 15743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:55:51.581754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44077 (local address 127.0.98.3:42665)
0929 17:55:51.582026 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0929 17:55:51.582084 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":93,"threads_started":1}
I20260929 17:55:52.286273 15744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:55:52.285851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44077 (local address 127.0.98.1:58823)
0929 17:55:52.286107 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0929 17:55:52.286167 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":72,"threads_started":1}
I20260929 17:55:52.415418 15745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:55:52.415007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44077 (local address 127.0.98.2:54765)
0929 17:55:52.415256 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0929 17:55:52.415314 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":87,"threads_started":1}
I20260929 17:55:52.583391 15746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:55:52.582952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44077 (local address 127.0.98.3:54885)
0929 17:55:52.583214 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0929 17:55:52.583285 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":79,"threads_started":1}
I20260929 17:55:53.287408 15747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:55:53.287007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44077 (local address 127.0.98.1:43287)
0929 17:55:53.287250 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0929 17:55:53.287306 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":79,"threads_started":1}
I20260929 17:55:53.416679 15748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:55:53.416208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44077 (local address 127.0.98.2:47355)
0929 17:55:53.416515 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0929 17:55:53.416581 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":80,"threads_started":1}
I20260929 17:55:53.584599 15749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:55:53.584185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44077 (local address 127.0.98.3:39505)
0929 17:55:53.584440 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0929 17:55:53.584502 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":77,"threads_started":1}
I20260929 17:55:53.670499 15750 raft_consensus.cc:482] T 34cb3888e81d468884bb1e9936a65cba P 40fc602f689f47a2afb4b5ddf158d470 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 17:55:53.670625 15750 raft_consensus.cc:504] T 34cb3888e81d468884bb1e9936a65cba P 40fc602f689f47a2afb4b5ddf158d470 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f14c4f2dc464bcb93ec6832e5ca1767" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35183 } } peers { permanent_uuid: "a5ea02aa22e3423e947ffd890f843ee7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39167 } } peers { permanent_uuid: "40fc602f689f47a2afb4b5ddf158d470" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37811 } }
I20260929 17:55:53.670998 15750 leader_election.cc:288] T 34cb3888e81d468884bb1e9936a65cba P 40fc602f689f47a2afb4b5ddf158d470 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3f14c4f2dc464bcb93ec6832e5ca1767 (127.0.98.2:35183), a5ea02aa22e3423e947ffd890f843ee7 (127.0.98.3:39167)
W20260929 17:55:53.671722 15322 leader_election.cc:341] T 34cb3888e81d468884bb1e9936a65cba P 40fc602f689f47a2afb4b5ddf158d470 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a5ea02aa22e3423e947ffd890f843ee7 (127.0.98.3:39167): Not found: Tablet not found: 34cb3888e81d468884bb1e9936a65cba
W20260929 17:55:53.671819 15320 leader_election.cc:341] T 34cb3888e81d468884bb1e9936a65cba P 40fc602f689f47a2afb4b5ddf158d470 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3f14c4f2dc464bcb93ec6832e5ca1767 (127.0.98.2:35183): Not found: Tablet not found: 34cb3888e81d468884bb1e9936a65cba
I20260929 17:55:53.671917 15320 leader_election.cc:302] T 34cb3888e81d468884bb1e9936a65cba P 40fc602f689f47a2afb4b5ddf158d470 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 40fc602f689f47a2afb4b5ddf158d470; no voters: 3f14c4f2dc464bcb93ec6832e5ca1767, a5ea02aa22e3423e947ffd890f843ee7
I20260929 17:55:53.672053 15750 raft_consensus.cc:2726] T 34cb3888e81d468884bb1e9936a65cba P 40fc602f689f47a2afb4b5ddf158d470 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 17:55:54.288520 15751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:55:54.288100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44077 (local address 127.0.98.1:36297)
0929 17:55:54.288367 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0929 17:55:54.288425 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":77,"threads_started":1}
I20260929 17:55:54.417910 15752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:55:54.417484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44077 (local address 127.0.98.2:52183)
0929 17:55:54.417763 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0929 17:55:54.417814 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":97,"threads_started":1}
I20260929 17:55:54.585721 15753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:55:54.585336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44077 (local address 127.0.98.3:56655)
0929 17:55:54.585578 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0929 17:55:54.585631 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":91,"threads_started":1}
I20260929 17:55:55.289891 15754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:55:55.289456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44077 (local address 127.0.98.1:54521)
0929 17:55:55.289701 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0929 17:55:55.289756 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":71,"threads_started":1}
I20260929 17:55:55.419186 15755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:55:55.418741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44077 (local address 127.0.98.2:48093)
0929 17:55:55.419036 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0929 17:55:55.419091 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":95,"threads_started":1}
I20260929 17:55:55.567914 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskup3Soe/build/debug/bin/kudu with pid 15270
I20260929 17:55:55.572932 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskup3Soe/build/debug/bin/kudu with pid 15437
I20260929 17:55:55.577566 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskup3Soe/build/debug/bin/kudu with pid 15571
2026-09-29T17:55:55Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (19261 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 17:55:55 dist-test-slave-53nh krb5kdc[15758](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 17:55:55 dist-test-slave-53nh krb5kdc[15758](info): set up 2 sockets
Sep 29 17:55:55 dist-test-slave-53nh krb5kdc[15758](info): commencing operation
krb5kdc: starting...
W20260929 17:55:56.524902 392 mini_kdc.cc:121] Time spent starting KDC: real 0.927s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/krb5kdc/test-user.keytab.
Sep 29 17:55:56 dist-test-slave-53nh krb5kdc[15758](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704556, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T17:55:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T17:55:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 17:55:56.637794 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskup3Soe/build/debug/bin/kudu
/tmp/dist-test-taskup3Soe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33709
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33709
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:55:56.744455 15771 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:55:56.744820 15771 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:55:56.744895 15771 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:55:56.750574 15771 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260929 17:55:56.750648 15771 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260929 17:55:56.750671 15771 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260929 17:55:56.750689 15771 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 17:55:56.750712 15771 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:55:56.750746 15771 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:55:56.750775 15771 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 17:55:56.750805 15771 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 17:55:56.756446 15771 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33709
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33709
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15771
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:55:56.757520 15771 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:55:56.758474 15771 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:55:56.766791 15777 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:55:56.766808 15776 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:55:56.766750 15779 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:55:56.766911 15771 server_base.cc:1034] running on GCE node
I20260929 17:55:56.767505 15771 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:55:56.768522 15771 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:55:56.769680 15771 hybrid_clock.cc:648] HybridClock initialized: now 1790704556769662 us; error 30 us; skew 500 ppm
Sep 29 17:55:56 dist-test-slave-53nh krb5kdc[15758](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704556, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:55:56.772220 15771 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 17:55:56.773046 15771 webserver.cc:460] Webserver started at http://127.0.98.62:43621/ using document root <none> and password file <none>
I20260929 17:55:56.773545 15771 fs_manager.cc:359] Metadata directory not provided
I20260929 17:55:56.773604 15771 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:55:56.773819 15771 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:55:56.775708 15771 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/minicluster-data/master-0/data/instance:
uuid: "842b44590ee242fca73034b1300da182"
format_stamp: "Formatted at 2026-09-29 17:55:56 on dist-test-slave-53nh"
server_key: "8021b0090bb6c1c80b9b0f788960a2e6"
server_key_iv: "f505f861ff6bb656c97c8657b4915be4"
server_key_version: "encryptionkey@0"
I20260929 17:55:56.776250 15771 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/minicluster-data/master-0/wal/instance:
uuid: "842b44590ee242fca73034b1300da182"
format_stamp: "Formatted at 2026-09-29 17:55:56 on dist-test-slave-53nh"
server_key: "8021b0090bb6c1c80b9b0f788960a2e6"
server_key_iv: "f505f861ff6bb656c97c8657b4915be4"
server_key_version: "encryptionkey@0"
I20260929 17:55:56.779706 15771 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260929 17:55:56.782006 15786 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:55:56.783100 15771 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 17:55:56.783206 15771 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/minicluster-data/master-0/wal
uuid: "842b44590ee242fca73034b1300da182"
format_stamp: "Formatted at 2026-09-29 17:55:56 on dist-test-slave-53nh"
server_key: "8021b0090bb6c1c80b9b0f788960a2e6"
server_key_iv: "f505f861ff6bb656c97c8657b4915be4"
server_key_version: "encryptionkey@0"
I20260929 17:55:56.783296 15771 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:55:56.805361 15771 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:55:56.808265 15771 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:55:56.808436 15771 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:55:56.814968 15771 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33709
I20260929 17:55:56.815237 15838 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33709 every 8 connection(s)
I20260929 17:55:56.815879 15771 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/minicluster-data/master-0/data/info.pb
I20260929 17:55:56.818513 15839 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:55:56.823678 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskup3Soe/build/debug/bin/kudu as pid 15771
I20260929 17:55:56.823789 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/minicluster-data/master-0/wal/instance
I20260929 17:55:56.823789 15839 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 842b44590ee242fca73034b1300da182: Bootstrap starting.
I20260929 17:55:56.824083 392 external_mini_cluster.cc:1428] Setting key aa0b9a23219cebe221b12552a34a88cc
I20260929 17:55:56.826198 15839 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 842b44590ee242fca73034b1300da182: Neither blocks nor log segments found. Creating new log.
I20260929 17:55:56.826870 15839 log.cc:824] T 00000000000000000000000000000000 P 842b44590ee242fca73034b1300da182: Log is configured to *not* fsync() on all Append() calls
I20260929 17:55:56.828689 15839 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 842b44590ee242fca73034b1300da182: No bootstrap required, opened a new log
Sep 29 17:55:56 dist-test-slave-53nh krb5kdc[15758](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704556, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 17:55:56.832612 15839 raft_consensus.cc:348] T 00000000000000000000000000000000 P 842b44590ee242fca73034b1300da182 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "842b44590ee242fca73034b1300da182" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33709 } }
I20260929 17:55:56.832849 15839 raft_consensus.cc:374] T 00000000000000000000000000000000 P 842b44590ee242fca73034b1300da182 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:55:56.832904 15839 raft_consensus.cc:729] T 00000000000000000000000000000000 P 842b44590ee242fca73034b1300da182 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 842b44590ee242fca73034b1300da182, State: Initialized, Role: FOLLOWER
I20260929 17:55:56.833377 15839 consensus_queue.cc:260] T 00000000000000000000000000000000 P 842b44590ee242fca73034b1300da182 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "842b44590ee242fca73034b1300da182" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33709 } }
I20260929 17:55:56.833546 15839 raft_consensus.cc:388] T 00000000000000000000000000000000 P 842b44590ee242fca73034b1300da182 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 17:55:56.833613 15839 raft_consensus.cc:482] T 00000000000000000000000000000000 P 842b44590ee242fca73034b1300da182 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 17:55:56.833709 15839 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 842b44590ee242fca73034b1300da182 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:55:56.834528 15839 raft_consensus.cc:504] T 00000000000000000000000000000000 P 842b44590ee242fca73034b1300da182 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "842b44590ee242fca73034b1300da182" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33709 } }
I20260929 17:55:56.834862 15839 leader_election.cc:302] T 00000000000000000000000000000000 P 842b44590ee242fca73034b1300da182 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 842b44590ee242fca73034b1300da182; no voters:
I20260929 17:55:56.835171 15839 leader_election.cc:288] T 00000000000000000000000000000000 P 842b44590ee242fca73034b1300da182 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 17:55:56.835280 15844 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 842b44590ee242fca73034b1300da182 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 17:55:56.835526 15844 raft_consensus.cc:686] T 00000000000000000000000000000000 P 842b44590ee242fca73034b1300da182 [term 1 LEADER]: Becoming Leader. State: Replica: 842b44590ee242fca73034b1300da182, State: Running, Role: LEADER
I20260929 17:55:56.835906 15844 consensus_queue.cc:237] T 00000000000000000000000000000000 P 842b44590ee242fca73034b1300da182 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "842b44590ee242fca73034b1300da182" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33709 } }
I20260929 17:55:56.836320 15839 sys_catalog.cc:564] T 00000000000000000000000000000000 P 842b44590ee242fca73034b1300da182 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 17:55:56.837509 15846 sys_catalog.cc:455] T 00000000000000000000000000000000 P 842b44590ee242fca73034b1300da182 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 842b44590ee242fca73034b1300da182. Latest consensus state: current_term: 1 leader_uuid: "842b44590ee242fca73034b1300da182" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "842b44590ee242fca73034b1300da182" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33709 } } }
I20260929 17:55:56.837620 15846 sys_catalog.cc:458] T 00000000000000000000000000000000 P 842b44590ee242fca73034b1300da182 [sys.catalog]: This master's current role is: LEADER
I20260929 17:55:56.837929 15845 sys_catalog.cc:455] T 00000000000000000000000000000000 P 842b44590ee242fca73034b1300da182 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "842b44590ee242fca73034b1300da182" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "842b44590ee242fca73034b1300da182" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33709 } } }
I20260929 17:55:56.838001 15845 sys_catalog.cc:458] T 00000000000000000000000000000000 P 842b44590ee242fca73034b1300da182 [sys.catalog]: This master's current role is: LEADER
I20260929 17:55:56.838495 15842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:55:56.825333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38306 (local address 127.0.98.62:33709)
0929 17:55:56.825724 (+ 391us) server_negotiation.cc:206] Beginning negotiation
0929 17:55:56.825739 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 17:55:56.825776 (+ 37us) server_negotiation.cc:407] Connection header received
0929 17:55:56.826408 (+ 632us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:55:56.826425 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:55:56.826747 (+ 322us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:55:56.827046 (+ 299us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:55:56.827771 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:55:56.828729 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:55:56.829599 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:55:56.830275 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:55:56.830440 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:55:56.832814 (+ 2374us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:55:56.832821 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:55:56.832837 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:55:56.832872 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:55:56.834572 (+ 1700us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:55:56.835051 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:55:56.835056 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:55:56.835060 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:55:56.835175 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:55:56.835502 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:55:56.835506 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:55:56.835507 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:55:56.836092 (+ 585us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:55:56.836269 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0929 17:55:56.836982 (+ 713us) server_negotiation.cc:299] Negotiation successful
0929 17:55:56.837309 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":263,"thread_start_us":120,"threads_started":1}
I20260929 17:55:56.838450 15854 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 17:55:56.841552 15854 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 17:55:56.846277 15854 catalog_manager.cc:1347] Generated new cluster ID: 92f56c901b26449fbe079d5de709dbdd
I20260929 17:55:56.846345 15854 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 17:55:56.859817 15854 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 17:55:56.860801 15854 catalog_manager.cc:1504] Loading token signing keys...
I20260929 17:55:56.876678 15854 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 842b44590ee242fca73034b1300da182: Generated new TSK 0
I20260929 17:55:56.877414 15854 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260929 17:55:56.907514 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskup3Soe/build/debug/bin/kudu
/tmp/dist-test-taskup3Soe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33709
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:55:57.018183 15865 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:55:57.018450 15865 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:55:57.018494 15865 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:55:57.023906 15865 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:55:57.024035 15865 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:55:57.024134 15865 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 17:55:57.031653 15865 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33709
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15865
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:55:57.032939 15865 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:55:57.033995 15865 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:55:57.043306 15870 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:55:57.043311 15873 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:55:57.043311 15871 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:55:57.043535 15865 server_base.cc:1034] running on GCE node
I20260929 17:55:57.043962 15865 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:55:57.044571 15865 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:55:57.045722 15865 hybrid_clock.cc:648] HybridClock initialized: now 1790704557045702 us; error 30 us; skew 500 ppm
Sep 29 17:55:57 dist-test-slave-53nh krb5kdc[15758](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704557, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:55:57.048355 15865 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260929 17:55:57.049247 15865 webserver.cc:460] Webserver started at http://127.0.98.1:43617/ using document root <none> and password file <none>
I20260929 17:55:57.049777 15865 fs_manager.cc:359] Metadata directory not provided
I20260929 17:55:57.049882 15865 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:55:57.050079 15865 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:55:57.052049 15865 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/minicluster-data/ts-0/data/instance:
uuid: "433a7f95a3694411b43c7a77d7fc93ea"
format_stamp: "Formatted at 2026-09-29 17:55:57 on dist-test-slave-53nh"
server_key: "232988f4455d1b6900d092f246158ce9"
server_key_iv: "c2e13854c1871df45f647fde8765fdfa"
server_key_version: "encryptionkey@0"
I20260929 17:55:57.052577 15865 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/minicluster-data/ts-0/wal/instance:
uuid: "433a7f95a3694411b43c7a77d7fc93ea"
format_stamp: "Formatted at 2026-09-29 17:55:57 on dist-test-slave-53nh"
server_key: "232988f4455d1b6900d092f246158ce9"
server_key_iv: "c2e13854c1871df45f647fde8765fdfa"
server_key_version: "encryptionkey@0"
I20260929 17:55:57.056212 15865 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260929 17:55:57.058583 15880 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:55:57.059759 15865 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 17:55:57.059875 15865 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/minicluster-data/ts-0/wal
uuid: "433a7f95a3694411b43c7a77d7fc93ea"
format_stamp: "Formatted at 2026-09-29 17:55:57 on dist-test-slave-53nh"
server_key: "232988f4455d1b6900d092f246158ce9"
server_key_iv: "c2e13854c1871df45f647fde8765fdfa"
server_key_version: "encryptionkey@0"
I20260929 17:55:57.059973 15865 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:55:57.074090 15865 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:55:57.077260 15865 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:55:57.077437 15865 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:55:57.078009 15865 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 17:55:57.078807 15865 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:55:57.078862 15865 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:55:57.078915 15865 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:55:57.078943 15865 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 17:55:57.087144 15865 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46395
I20260929 17:55:57.087182 15993 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46395 every 8 connection(s)
I20260929 17:55:57.088027 15865 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/minicluster-data/ts-0/data/info.pb
I20260929 17:55:57.093593 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskup3Soe/build/debug/bin/kudu as pid 15865
I20260929 17:55:57.093711 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/minicluster-data/ts-0/wal/instance
I20260929 17:55:57.093989 392 external_mini_cluster.cc:1428] Setting key 0903a2de6f7731432afab8d86c3fa6c3
Sep 29 17:55:57 dist-test-slave-53nh krb5kdc[15758](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704557, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 17:55:57.102222 15842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:55:57.089830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47763 (local address 127.0.98.62:33709)
0929 17:55:57.089948 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0929 17:55:57.089951 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 17:55:57.090722 (+ 771us) server_negotiation.cc:407] Connection header received
0929 17:55:57.092000 (+ 1278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:55:57.092003 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:55:57.092050 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:55:57.092123 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:55:57.093410 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:55:57.094198 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:55:57.095562 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:55:57.095920 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:55:57.096016 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:55:57.098544 (+ 2528us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:55:57.098547 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:55:57.098549 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:55:57.098572 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:55:57.099949 (+ 1377us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:55:57.100422 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:55:57.100425 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:55:57.100427 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:55:57.100503 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:55:57.100906 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:55:57.100909 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:55:57.100912 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:55:57.101184 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:55:57.101264 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0929 17:55:57.101918 (+ 654us) server_negotiation.cc:299] Negotiation successful
0929 17:55:57.102053 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260929 17:55:57.103065 15996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:55:57.090138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33709 (local address 127.0.98.1:47763)
0929 17:55:57.090569 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0929 17:55:57.090616 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 17:55:57.091755 (+ 1139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:55:57.092271 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:55:57.092277 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:55:57.092647 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:55:57.093249 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:55:57.093264 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:55:57.094345 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:55:57.094348 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:55:57.095458 (+ 1110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:55:57.095468 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:55:57.096060 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:55:57.096064 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:55:57.096209 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:55:57.096781 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:55:57.096802 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:55:57.098390 (+ 1588us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:55:57.100054 (+ 1664us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:55:57.100060 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:55:57.100075 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:55:57.100325 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:55:57.100583 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:55:57.100586 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:55:57.100588 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:55:57.100780 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:55:57.101351 (+ 571us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:55:57.101358 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:55:57.101681 (+ 323us) client_negotiation.cc:769] Sending connection context
0929 17:55:57.101860 (+ 179us) client_negotiation.cc:240] Negotiation successful
0929 17:55:57.102197 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":108,"threads_started":1}
I20260929 17:55:57.104362 15994 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33709
I20260929 17:55:57.104610 15994 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260929 17:55:57.105119 15994 heartbeater.cc:511] Master 127.0.98.62:33709 requested a full tablet report, sending...
I20260929 17:55:57.106855 15803 ts_manager.cc:194] Registered new tserver with Master: 433a7f95a3694411b43c7a77d7fc93ea (127.0.98.1:46395)
I20260929 17:55:57.108407 15803 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:47763
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260929 17:55:57.120833 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskup3Soe/build/debug/bin/kudu
/tmp/dist-test-taskup3Soe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33709
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:55:57.226490 15999 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:55:57.226747 15999 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:55:57.226788 15999 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:55:57.230722 15999 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:55:57.230777 15999 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:55:57.230846 15999 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 17:55:57.236104 15999 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33709
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15999
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:55:57.237181 15999 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:55:57.238202 15999 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:55:57.247747 16004 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:55:57.247948 15999 server_base.cc:1034] running on GCE node
W20260929 17:55:57.247747 16005 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:55:57.247748 16007 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:55:57.248471 15999 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:55:57.249168 15999 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:55:57.250322 15999 hybrid_clock.cc:648] HybridClock initialized: now 1790704557250303 us; error 32 us; skew 500 ppm
Sep 29 17:55:57 dist-test-slave-53nh krb5kdc[15758](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704557, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:55:57.253134 15999 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260929 17:55:57.254045 15999 webserver.cc:460] Webserver started at http://127.0.98.2:35103/ using document root <none> and password file <none>
I20260929 17:55:57.254616 15999 fs_manager.cc:359] Metadata directory not provided
I20260929 17:55:57.254681 15999 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:55:57.254870 15999 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:55:57.256935 15999 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/minicluster-data/ts-1/data/instance:
uuid: "b499dd73acdb48028230a6e4f02bd2d4"
format_stamp: "Formatted at 2026-09-29 17:55:57 on dist-test-slave-53nh"
server_key: "d2473c82904daeddcebaac9867c45400"
server_key_iv: "1943549757123a78e76af5f53abfd36d"
server_key_version: "encryptionkey@0"
I20260929 17:55:57.257525 15999 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/minicluster-data/ts-1/wal/instance:
uuid: "b499dd73acdb48028230a6e4f02bd2d4"
format_stamp: "Formatted at 2026-09-29 17:55:57 on dist-test-slave-53nh"
server_key: "d2473c82904daeddcebaac9867c45400"
server_key_iv: "1943549757123a78e76af5f53abfd36d"
server_key_version: "encryptionkey@0"
I20260929 17:55:57.261147 15999 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 17:55:57.263545 16014 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:55:57.264815 15999 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 17:55:57.264921 15999 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/minicluster-data/ts-1/wal
uuid: "b499dd73acdb48028230a6e4f02bd2d4"
format_stamp: "Formatted at 2026-09-29 17:55:57 on dist-test-slave-53nh"
server_key: "d2473c82904daeddcebaac9867c45400"
server_key_iv: "1943549757123a78e76af5f53abfd36d"
server_key_version: "encryptionkey@0"
I20260929 17:55:57.265039 15999 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:55:57.283701 15999 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:55:57.286724 15999 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:55:57.286887 15999 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:55:57.287431 15999 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 17:55:57.288240 15999 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:55:57.288285 15999 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:55:57.288321 15999 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:55:57.288341 15999 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 17:55:57.296440 15999 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33451
I20260929 17:55:57.296474 16127 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33451 every 8 connection(s)
I20260929 17:55:57.297381 15999 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/minicluster-data/ts-1/data/info.pb
I20260929 17:55:57.306716 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskup3Soe/build/debug/bin/kudu as pid 15999
Sep 29 17:55:57 dist-test-slave-53nh krb5kdc[15758](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704557, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 17:55:57.306813 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/minicluster-data/ts-1/wal/instance
I20260929 17:55:57.307047 392 external_mini_cluster.cc:1428] Setting key f86d16a8ba6784f7e49086b24dee7e2a
I20260929 17:55:57.311034 15842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:55:57.299215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39377 (local address 127.0.98.62:33709)
0929 17:55:57.299358 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0929 17:55:57.299371 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 17:55:57.300381 (+ 1010us) server_negotiation.cc:407] Connection header received
0929 17:55:57.301708 (+ 1327us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:55:57.301711 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:55:57.301768 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:55:57.301843 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:55:57.303033 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:55:57.303589 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:55:57.304688 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:55:57.304920 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:55:57.305039 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:55:57.307389 (+ 2350us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:55:57.307393 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:55:57.307396 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:55:57.307425 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:55:57.308832 (+ 1407us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:55:57.309399 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:55:57.309402 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:55:57.309403 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:55:57.309479 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:55:57.309926 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:55:57.309930 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:55:57.309931 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:55:57.310178 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:55:57.310278 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0929 17:55:57.310752 (+ 474us) server_negotiation.cc:299] Negotiation successful
0929 17:55:57.310877 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 17:55:57.311812 16130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:55:57.299609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33709 (local address 127.0.98.2:39377)
0929 17:55:57.300187 (+ 578us) negotiation.cc:107] Waiting for socket to connect
0929 17:55:57.300235 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0929 17:55:57.301493 (+ 1258us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:55:57.302000 (+ 507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:55:57.302007 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:55:57.302401 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:55:57.302889 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:55:57.302903 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:55:57.303689 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:55:57.303692 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:55:57.304570 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:55:57.304578 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:55:57.305075 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:55:57.305078 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:55:57.305216 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:55:57.305760 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:55:57.305776 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:55:57.307205 (+ 1429us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:55:57.308938 (+ 1733us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:55:57.308944 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:55:57.308960 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:55:57.309303 (+ 343us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:55:57.309553 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:55:57.309556 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:55:57.309558 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:55:57.309729 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:55:57.310277 (+ 548us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:55:57.310282 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:55:57.310506 (+ 224us) client_negotiation.cc:769] Sending connection context
0929 17:55:57.310673 (+ 167us) client_negotiation.cc:240] Negotiation successful
0929 17:55:57.310999 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":166,"threads_started":1}
I20260929 17:55:57.312894 16128 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33709
I20260929 17:55:57.313145 16128 heartbeater.cc:464] Registering TS with master...
I20260929 17:55:57.313604 16128 heartbeater.cc:511] Master 127.0.98.62:33709 requested a full tablet report, sending...
I20260929 17:55:57.314591 15803 ts_manager.cc:194] Registered new tserver with Master: b499dd73acdb48028230a6e4f02bd2d4 (127.0.98.2:33451)
I20260929 17:55:57.315351 15803 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:39377
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260929 17:55:57.332374 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskup3Soe/build/debug/bin/kudu
/tmp/dist-test-taskup3Soe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33709
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:55:57.442402 16133 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:55:57.442663 16133 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:55:57.442713 16133 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:55:57.446940 16133 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:55:57.447010 16133 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:55:57.447079 16133 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 17:55:57.452317 16133 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33709
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16133
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:55:57.453328 16133 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:55:57.454315 16133 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:55:57.463377 16139 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:55:57.463495 16141 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:55:57.463428 16138 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:55:57.463406 16133 server_base.cc:1034] running on GCE node
I20260929 17:55:57.463975 16133 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:55:57.464632 16133 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:55:57.465780 16133 hybrid_clock.cc:648] HybridClock initialized: now 1790704557465768 us; error 26 us; skew 500 ppm
Sep 29 17:55:57 dist-test-slave-53nh krb5kdc[15758](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704557, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:55:57.468387 16133 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260929 17:55:57.469300 16133 webserver.cc:460] Webserver started at http://127.0.98.3:46089/ using document root <none> and password file <none>
I20260929 17:55:57.469805 16133 fs_manager.cc:359] Metadata directory not provided
I20260929 17:55:57.469859 16133 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:55:57.470064 16133 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:55:57.471968 16133 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/minicluster-data/ts-2/data/instance:
uuid: "e8a734f480b34296ac1bb50572558f10"
format_stamp: "Formatted at 2026-09-29 17:55:57 on dist-test-slave-53nh"
server_key: "43d0aeb3bdff69a5cbc3dca0926f9313"
server_key_iv: "d6083adc1f9f419fab5de7ef217989b3"
server_key_version: "encryptionkey@0"
I20260929 17:55:57.472462 16133 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/minicluster-data/ts-2/wal/instance:
uuid: "e8a734f480b34296ac1bb50572558f10"
format_stamp: "Formatted at 2026-09-29 17:55:57 on dist-test-slave-53nh"
server_key: "43d0aeb3bdff69a5cbc3dca0926f9313"
server_key_iv: "d6083adc1f9f419fab5de7ef217989b3"
server_key_version: "encryptionkey@0"
I20260929 17:55:57.475932 16133 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260929 17:55:57.478277 16148 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:55:57.479427 16133 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 17:55:57.479537 16133 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/minicluster-data/ts-2/wal
uuid: "e8a734f480b34296ac1bb50572558f10"
format_stamp: "Formatted at 2026-09-29 17:55:57 on dist-test-slave-53nh"
server_key: "43d0aeb3bdff69a5cbc3dca0926f9313"
server_key_iv: "d6083adc1f9f419fab5de7ef217989b3"
server_key_version: "encryptionkey@0"
I20260929 17:55:57.479627 16133 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:55:57.500252 16133 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:55:57.503232 16133 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:55:57.503405 16133 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:55:57.503979 16133 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 17:55:57.504750 16133 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:55:57.504796 16133 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:55:57.504846 16133 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:55:57.504874 16133 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 17:55:57.513020 16133 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34933
I20260929 17:55:57.513073 16261 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34933 every 8 connection(s)
I20260929 17:55:57.513871 16133 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/minicluster-data/ts-2/data/info.pb
I20260929 17:55:57.518445 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskup3Soe/build/debug/bin/kudu as pid 16133
I20260929 17:55:57.518548 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/minicluster-data/ts-2/wal/instance
I20260929 17:55:57.518775 392 external_mini_cluster.cc:1428] Setting key 69fa849997d5438fe1e9f68ab845b939
Sep 29 17:55:57 dist-test-slave-53nh krb5kdc[15758](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704557, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 17:55:57.526540 15842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:55:57.515583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35279 (local address 127.0.98.62:33709)
0929 17:55:57.515725 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0929 17:55:57.515729 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:55:57.516550 (+ 821us) server_negotiation.cc:407] Connection header received
0929 17:55:57.517851 (+ 1301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:55:57.517854 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:55:57.517918 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:55:57.517988 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:55:57.519163 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:55:57.519743 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:55:57.520862 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:55:57.521123 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:55:57.521205 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:55:57.523522 (+ 2317us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:55:57.523526 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:55:57.523529 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:55:57.523553 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:55:57.524675 (+ 1122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:55:57.525164 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:55:57.525168 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:55:57.525170 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:55:57.525244 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:55:57.525543 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:55:57.525546 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:55:57.525548 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:55:57.525718 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:55:57.525801 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0929 17:55:57.526263 (+ 462us) server_negotiation.cc:299] Negotiation successful
0929 17:55:57.526383 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 17:55:57.527292 16264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:55:57.515956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33709 (local address 127.0.98.3:35279)
0929 17:55:57.516376 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0929 17:55:57.516447 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0929 17:55:57.517650 (+ 1203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:55:57.518162 (+ 512us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:55:57.518169 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:55:57.518525 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:55:57.519030 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:55:57.519045 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:55:57.519856 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:55:57.519859 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:55:57.520754 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:55:57.520763 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:55:57.521261 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:55:57.521265 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:55:57.521420 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:55:57.521898 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:55:57.521914 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:55:57.523345 (+ 1431us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:55:57.524776 (+ 1431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:55:57.524781 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:55:57.524797 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:55:57.525073 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:55:57.525319 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:55:57.525322 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:55:57.525324 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:55:57.525460 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:55:57.525812 (+ 352us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:55:57.525818 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:55:57.526047 (+ 229us) client_negotiation.cc:769] Sending connection context
0929 17:55:57.526206 (+ 159us) client_negotiation.cc:240] Negotiation successful
0929 17:55:57.526520 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":116,"threads_started":1}
I20260929 17:55:57.528259 16262 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33709
I20260929 17:55:57.528478 16262 heartbeater.cc:464] Registering TS with master...
I20260929 17:55:57.528939 16262 heartbeater.cc:511] Master 127.0.98.62:33709 requested a full tablet report, sending...
I20260929 17:55:57.529862 15803 ts_manager.cc:194] Registered new tserver with Master: e8a734f480b34296ac1bb50572558f10 (127.0.98.3:34933)
I20260929 17:55:57.530369 15803 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:35279
I20260929 17:55:57.531998 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 17:55:57.540333 15842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:55:57.533740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38312 (local address 127.0.98.62:33709)
0929 17:55:57.533853 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0929 17:55:57.533858 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:55:57.534042 (+ 184us) server_negotiation.cc:407] Connection header received
0929 17:55:57.534159 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:55:57.534162 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:55:57.534201 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:55:57.534258 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:55:57.534875 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:55:57.535365 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:55:57.536198 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:55:57.536417 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:55:57.536496 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:55:57.537713 (+ 1217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:55:57.537717 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:55:57.537719 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:55:57.537742 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:55:57.538897 (+ 1155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:55:57.539245 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:55:57.539248 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:55:57.539250 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:55:57.539327 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:55:57.539553 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:55:57.539557 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:55:57.539558 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:55:57.539754 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:55:57.539867 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0929 17:55:57.540082 (+ 215us) server_negotiation.cc:299] Negotiation successful
0929 17:55:57.540183 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 17:55:57.542830 15803 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38312:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 17:55:57.545814 15803 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 17:55:57.559825 16275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:55:57.554487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34933 (local address 127.0.0.1:34608)
0929 17:55:57.555165 (+ 678us) negotiation.cc:107] Waiting for socket to connect
0929 17:55:57.555177 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 17:55:57.555245 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:55:57.555976 (+ 731us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:55:57.555979 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:55:57.555998 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:55:57.556079 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:55:57.556084 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:55:57.557054 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:55:57.557057 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:55:57.558279 (+ 1222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:55:57.558290 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:55:57.559279 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:55:57.559283 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:55:57.559381 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:55:57.559417 (+ 36us) client_negotiation.cc:769] Sending connection context
0929 17:55:57.559524 (+ 107us) client_negotiation.cc:240] Negotiation successful
0929 17:55:57.559652 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":625,"mutex_wait_us":1,"thread_start_us":75,"threads_started":1}
I20260929 17:55:57.559849 16273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:55:57.553797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33451 (local address 127.0.0.1:37450)
0929 17:55:57.554148 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0929 17:55:57.554204 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0929 17:55:57.554514 (+ 310us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:55:57.555801 (+ 1287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:55:57.555807 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:55:57.555829 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:55:57.556027 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:55:57.556036 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:55:57.557529 (+ 1493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:55:57.557532 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:55:57.558595 (+ 1063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:55:57.558604 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:55:57.559539 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:55:57.559542 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:55:57.559628 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:55:57.559647 (+ 19us) client_negotiation.cc:769] Sending connection context
0929 17:55:57.559699 (+ 52us) client_negotiation.cc:240] Negotiation successful
0929 17:55:57.559753 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"mutex_wait_us":56,"thread_start_us":79,"threads_started":1}
I20260929 17:55:57.559989 16278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:55:57.554953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37450 (local address 127.0.98.2:33451)
0929 17:55:57.555318 (+ 365us) server_negotiation.cc:206] Beginning negotiation
0929 17:55:57.555331 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 17:55:57.555354 (+ 23us) server_negotiation.cc:407] Connection header received
0929 17:55:57.555427 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:55:57.555433 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:55:57.555635 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:55:57.555793 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:55:57.556157 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:55:57.557365 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:55:57.558704 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:55:57.559424 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:55:57.559533 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:55:57.559669 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0929 17:55:57.559729 (+ 60us) server_negotiation.cc:299] Negotiation successful
0929 17:55:57.559828 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282,"thread_start_us":125,"threads_started":1}
I20260929 17:55:57.560194 16274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:55:57.554202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46395 (local address 127.0.0.1:60886)
0929 17:55:57.554900 (+ 698us) negotiation.cc:107] Waiting for socket to connect
0929 17:55:57.554913 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 17:55:57.554995 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:55:57.555868 (+ 873us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:55:57.555871 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:55:57.555883 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:55:57.556035 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:55:57.556042 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:55:57.557501 (+ 1459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:55:57.557504 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:55:57.558828 (+ 1324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:55:57.558837 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:55:57.559835 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:55:57.559838 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:55:57.559933 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:55:57.559956 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 17:55:57.560004 (+ 48us) client_negotiation.cc:240] Negotiation successful
0929 17:55:57.560068 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":639,"mutex_wait_us":217,"thread_start_us":108,"threads_started":1}
I20260929 17:55:57.560477 16277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:55:57.554895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34608 (local address 127.0.98.3:34933)
0929 17:55:57.555318 (+ 423us) server_negotiation.cc:206] Beginning negotiation
0929 17:55:57.555331 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 17:55:57.555354 (+ 23us) server_negotiation.cc:407] Connection header received
0929 17:55:57.555427 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:55:57.555433 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:55:57.555653 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:55:57.555793 (+ 140us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:55:57.556187 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:55:57.556919 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:55:57.558430 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:55:57.559174 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:55:57.559267 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:55:57.559432 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0929 17:55:57.560241 (+ 809us) server_negotiation.cc:299] Negotiation successful
0929 17:55:57.560345 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":121,"threads_started":1}
I20260929 17:55:57.561326 16276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:55:57.554623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60886 (local address 127.0.98.1:46395)
0929 17:55:57.555489 (+ 866us) server_negotiation.cc:206] Beginning negotiation
0929 17:55:57.555501 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 17:55:57.555524 (+ 23us) server_negotiation.cc:407] Connection header received
0929 17:55:57.555585 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:55:57.555589 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:55:57.555736 (+ 147us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:55:57.555985 (+ 249us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:55:57.556136 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:55:57.557365 (+ 1229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:55:57.558945 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:55:57.559711 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:55:57.560878 (+ 1167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:55:57.561053 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0929 17:55:57.561101 (+ 48us) server_negotiation.cc:299] Negotiation successful
0929 17:55:57.561194 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":766,"thread_start_us":133,"threads_started":1}
I20260929 17:55:57.563117 16062 tablet_service.cc:1467] Processing CreateTablet for tablet 209ecdb78af240aeb4ae8a20046611ee (DEFAULT_TABLE table=test-table [id=8cbc0e9f803d411a9f7c70253d10ba55]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 17:55:57.563371 16196 tablet_service.cc:1467] Processing CreateTablet for tablet 209ecdb78af240aeb4ae8a20046611ee (DEFAULT_TABLE table=test-table [id=8cbc0e9f803d411a9f7c70253d10ba55]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 17:55:57.563755 15928 tablet_service.cc:1467] Processing CreateTablet for tablet 209ecdb78af240aeb4ae8a20046611ee (DEFAULT_TABLE table=test-table [id=8cbc0e9f803d411a9f7c70253d10ba55]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 17:55:57.564354 16062 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 209ecdb78af240aeb4ae8a20046611ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:55:57.564383 16196 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 209ecdb78af240aeb4ae8a20046611ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:55:57.564611 15928 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 209ecdb78af240aeb4ae8a20046611ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:55:57.570341 16279 tablet_bootstrap.cc:492] T 209ecdb78af240aeb4ae8a20046611ee P b499dd73acdb48028230a6e4f02bd2d4: Bootstrap starting.
I20260929 17:55:57.571326 16280 tablet_bootstrap.cc:492] T 209ecdb78af240aeb4ae8a20046611ee P e8a734f480b34296ac1bb50572558f10: Bootstrap starting.
I20260929 17:55:57.572369 16281 tablet_bootstrap.cc:492] T 209ecdb78af240aeb4ae8a20046611ee P 433a7f95a3694411b43c7a77d7fc93ea: Bootstrap starting.
I20260929 17:55:57.573354 16280 tablet_bootstrap.cc:654] T 209ecdb78af240aeb4ae8a20046611ee P e8a734f480b34296ac1bb50572558f10: Neither blocks nor log segments found. Creating new log.
I20260929 17:55:57.573392 16279 tablet_bootstrap.cc:654] T 209ecdb78af240aeb4ae8a20046611ee P b499dd73acdb48028230a6e4f02bd2d4: Neither blocks nor log segments found. Creating new log.
I20260929 17:55:57.574018 16281 tablet_bootstrap.cc:654] T 209ecdb78af240aeb4ae8a20046611ee P 433a7f95a3694411b43c7a77d7fc93ea: Neither blocks nor log segments found. Creating new log.
I20260929 17:55:57.574237 16279 log.cc:824] T 209ecdb78af240aeb4ae8a20046611ee P b499dd73acdb48028230a6e4f02bd2d4: Log is configured to *not* fsync() on all Append() calls
I20260929 17:55:57.574456 16280 log.cc:824] T 209ecdb78af240aeb4ae8a20046611ee P e8a734f480b34296ac1bb50572558f10: Log is configured to *not* fsync() on all Append() calls
I20260929 17:55:57.575037 16281 log.cc:824] T 209ecdb78af240aeb4ae8a20046611ee P 433a7f95a3694411b43c7a77d7fc93ea: Log is configured to *not* fsync() on all Append() calls
I20260929 17:55:57.576107 16279 tablet_bootstrap.cc:492] T 209ecdb78af240aeb4ae8a20046611ee P b499dd73acdb48028230a6e4f02bd2d4: No bootstrap required, opened a new log
I20260929 17:55:57.576185 16280 tablet_bootstrap.cc:492] T 209ecdb78af240aeb4ae8a20046611ee P e8a734f480b34296ac1bb50572558f10: No bootstrap required, opened a new log
I20260929 17:55:57.576285 16279 ts_tablet_manager.cc:1397] T 209ecdb78af240aeb4ae8a20046611ee P b499dd73acdb48028230a6e4f02bd2d4: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 17:55:57.576370 16280 ts_tablet_manager.cc:1397] T 209ecdb78af240aeb4ae8a20046611ee P e8a734f480b34296ac1bb50572558f10: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260929 17:55:57.576735 16281 tablet_bootstrap.cc:492] T 209ecdb78af240aeb4ae8a20046611ee P 433a7f95a3694411b43c7a77d7fc93ea: No bootstrap required, opened a new log
I20260929 17:55:57.576902 16281 ts_tablet_manager.cc:1397] T 209ecdb78af240aeb4ae8a20046611ee P 433a7f95a3694411b43c7a77d7fc93ea: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260929 17:55:57.579427 16279 raft_consensus.cc:348] T 209ecdb78af240aeb4ae8a20046611ee P b499dd73acdb48028230a6e4f02bd2d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b499dd73acdb48028230a6e4f02bd2d4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33451 } } peers { permanent_uuid: "e8a734f480b34296ac1bb50572558f10" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34933 } } peers { permanent_uuid: "433a7f95a3694411b43c7a77d7fc93ea" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46395 } }
I20260929 17:55:57.579649 16279 raft_consensus.cc:374] T 209ecdb78af240aeb4ae8a20046611ee P b499dd73acdb48028230a6e4f02bd2d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:55:57.579728 16279 raft_consensus.cc:729] T 209ecdb78af240aeb4ae8a20046611ee P b499dd73acdb48028230a6e4f02bd2d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b499dd73acdb48028230a6e4f02bd2d4, State: Initialized, Role: FOLLOWER
I20260929 17:55:57.580139 16279 consensus_queue.cc:260] T 209ecdb78af240aeb4ae8a20046611ee P b499dd73acdb48028230a6e4f02bd2d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b499dd73acdb48028230a6e4f02bd2d4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33451 } } peers { permanent_uuid: "e8a734f480b34296ac1bb50572558f10" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34933 } } peers { permanent_uuid: "433a7f95a3694411b43c7a77d7fc93ea" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46395 } }
I20260929 17:55:57.580027 16280 raft_consensus.cc:348] T 209ecdb78af240aeb4ae8a20046611ee P e8a734f480b34296ac1bb50572558f10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b499dd73acdb48028230a6e4f02bd2d4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33451 } } peers { permanent_uuid: "e8a734f480b34296ac1bb50572558f10" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34933 } } peers { permanent_uuid: "433a7f95a3694411b43c7a77d7fc93ea" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46395 } }
I20260929 17:55:57.580266 16280 raft_consensus.cc:374] T 209ecdb78af240aeb4ae8a20046611ee P e8a734f480b34296ac1bb50572558f10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:55:57.580368 16280 raft_consensus.cc:729] T 209ecdb78af240aeb4ae8a20046611ee P e8a734f480b34296ac1bb50572558f10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8a734f480b34296ac1bb50572558f10, State: Initialized, Role: FOLLOWER
I20260929 17:55:57.580415 16281 raft_consensus.cc:348] T 209ecdb78af240aeb4ae8a20046611ee P 433a7f95a3694411b43c7a77d7fc93ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b499dd73acdb48028230a6e4f02bd2d4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33451 } } peers { permanent_uuid: "e8a734f480b34296ac1bb50572558f10" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34933 } } peers { permanent_uuid: "433a7f95a3694411b43c7a77d7fc93ea" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46395 } }
I20260929 17:55:57.580704 16281 raft_consensus.cc:374] T 209ecdb78af240aeb4ae8a20046611ee P 433a7f95a3694411b43c7a77d7fc93ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:55:57.580768 16128 heartbeater.cc:503] Master 127.0.98.62:33709 was elected leader, sending a full tablet report...
I20260929 17:55:57.580790 16281 raft_consensus.cc:729] T 209ecdb78af240aeb4ae8a20046611ee P 433a7f95a3694411b43c7a77d7fc93ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 433a7f95a3694411b43c7a77d7fc93ea, State: Initialized, Role: FOLLOWER
I20260929 17:55:57.580878 16280 consensus_queue.cc:260] T 209ecdb78af240aeb4ae8a20046611ee P e8a734f480b34296ac1bb50572558f10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b499dd73acdb48028230a6e4f02bd2d4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33451 } } peers { permanent_uuid: "e8a734f480b34296ac1bb50572558f10" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34933 } } peers { permanent_uuid: "433a7f95a3694411b43c7a77d7fc93ea" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46395 } }
I20260929 17:55:57.581110 16279 ts_tablet_manager.cc:1428] T 209ecdb78af240aeb4ae8a20046611ee P b499dd73acdb48028230a6e4f02bd2d4: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260929 17:55:57.581313 16281 consensus_queue.cc:260] T 209ecdb78af240aeb4ae8a20046611ee P 433a7f95a3694411b43c7a77d7fc93ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b499dd73acdb48028230a6e4f02bd2d4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33451 } } peers { permanent_uuid: "e8a734f480b34296ac1bb50572558f10" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34933 } } peers { permanent_uuid: "433a7f95a3694411b43c7a77d7fc93ea" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46395 } }
I20260929 17:55:57.581743 16262 heartbeater.cc:503] Master 127.0.98.62:33709 was elected leader, sending a full tablet report...
I20260929 17:55:57.582017 16280 ts_tablet_manager.cc:1428] T 209ecdb78af240aeb4ae8a20046611ee P e8a734f480b34296ac1bb50572558f10: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260929 17:55:57.582155 16281 ts_tablet_manager.cc:1428] T 209ecdb78af240aeb4ae8a20046611ee P 433a7f95a3694411b43c7a77d7fc93ea: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260929 17:55:57.582445 15994 heartbeater.cc:503] Master 127.0.98.62:33709 was elected leader, sending a full tablet report...
W20260929 17:55:57.589075 15995 tablet.cc:2369] T 209ecdb78af240aeb4ae8a20046611ee P 433a7f95a3694411b43c7a77d7fc93ea: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 17:55:57.589349 16285 raft_consensus.cc:482] T 209ecdb78af240aeb4ae8a20046611ee P b499dd73acdb48028230a6e4f02bd2d4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 17:55:57.589475 16285 raft_consensus.cc:504] T 209ecdb78af240aeb4ae8a20046611ee P b499dd73acdb48028230a6e4f02bd2d4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b499dd73acdb48028230a6e4f02bd2d4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33451 } } peers { permanent_uuid: "e8a734f480b34296ac1bb50572558f10" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34933 } } peers { permanent_uuid: "433a7f95a3694411b43c7a77d7fc93ea" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46395 } }
I20260929 17:55:57.590454 16285 leader_election.cc:288] T 209ecdb78af240aeb4ae8a20046611ee P b499dd73acdb48028230a6e4f02bd2d4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e8a734f480b34296ac1bb50572558f10 (127.0.98.3:34933), 433a7f95a3694411b43c7a77d7fc93ea (127.0.98.1:46395)
I20260929 17:55:57.594200 16277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:55:57.590704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36949 (local address 127.0.98.3:34933)
0929 17:55:57.590825 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0929 17:55:57.590830 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:55:57.590909 (+ 79us) server_negotiation.cc:407] Connection header received
0929 17:55:57.591021 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:55:57.591024 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:55:57.591063 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:55:57.591119 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:55:57.591360 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:55:57.591836 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:55:57.592970 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:55:57.593611 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:55:57.593712 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:55:57.593771 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0929 17:55:57.594017 (+ 246us) server_negotiation.cc:299] Negotiation successful
0929 17:55:57.594083 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260929 17:55:57.594200 16276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:55:57.590841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42127 (local address 127.0.98.1:46395)
0929 17:55:57.590944 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0929 17:55:57.590949 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:55:57.591057 (+ 108us) server_negotiation.cc:407] Connection header received
0929 17:55:57.591158 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:55:57.591161 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:55:57.591199 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:55:57.591258 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:55:57.591481 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:55:57.591955 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:55:57.593030 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:55:57.593611 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:55:57.593712 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:55:57.593771 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0929 17:55:57.594017 (+ 246us) server_negotiation.cc:299] Negotiation successful
0929 17:55:57.594088 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260929 17:55:57.594218 16130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:55:57.590639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34933 (local address 127.0.98.2:36949)
0929 17:55:57.590797 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0929 17:55:57.590842 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 17:55:57.590929 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:55:57.591136 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:55:57.591139 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:55:57.591153 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:55:57.591253 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:55:57.591258 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:55:57.591928 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:55:57.591931 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:55:57.592865 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:55:57.592874 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:55:57.593714 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:55:57.593718 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:55:57.593871 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:55:57.593916 (+ 45us) client_negotiation.cc:769] Sending connection context
0929 17:55:57.593982 (+ 66us) client_negotiation.cc:240] Negotiation successful
0929 17:55:57.594065 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":87,"mutex_wait_us":107}
I20260929 17:55:57.594231 16288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:55:57.590670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46395 (local address 127.0.98.2:42127)
0929 17:55:57.590980 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0929 17:55:57.590991 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 17:55:57.591084 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:55:57.591273 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:55:57.591275 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:55:57.591289 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:55:57.591395 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:55:57.591400 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:55:57.592046 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:55:57.592049 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:55:57.592879 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:55:57.592889 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:55:57.593714 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:55:57.593717 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:55:57.593877 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:55:57.593910 (+ 33us) client_negotiation.cc:769] Sending connection context
0929 17:55:57.593982 (+ 72us) client_negotiation.cc:240] Negotiation successful
0929 17:55:57.594065 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"mutex_wait_us":23,"thread_start_us":106,"threads_started":1}
I20260929 17:55:57.594806 15948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "209ecdb78af240aeb4ae8a20046611ee" candidate_uuid: "b499dd73acdb48028230a6e4f02bd2d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "433a7f95a3694411b43c7a77d7fc93ea" is_pre_election: true
I20260929 17:55:57.594878 16216 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "209ecdb78af240aeb4ae8a20046611ee" candidate_uuid: "b499dd73acdb48028230a6e4f02bd2d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8a734f480b34296ac1bb50572558f10" is_pre_election: true
I20260929 17:55:57.595067 15948 raft_consensus.cc:2454] T 209ecdb78af240aeb4ae8a20046611ee P 433a7f95a3694411b43c7a77d7fc93ea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b499dd73acdb48028230a6e4f02bd2d4 in term 0.
I20260929 17:55:57.595103 16216 raft_consensus.cc:2454] T 209ecdb78af240aeb4ae8a20046611ee P e8a734f480b34296ac1bb50572558f10 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b499dd73acdb48028230a6e4f02bd2d4 in term 0.
I20260929 17:55:57.595465 16016 leader_election.cc:302] T 209ecdb78af240aeb4ae8a20046611ee P b499dd73acdb48028230a6e4f02bd2d4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 433a7f95a3694411b43c7a77d7fc93ea, b499dd73acdb48028230a6e4f02bd2d4; no voters:
I20260929 17:55:57.595679 16285 raft_consensus.cc:2781] T 209ecdb78af240aeb4ae8a20046611ee P b499dd73acdb48028230a6e4f02bd2d4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 17:55:57.595757 16285 raft_consensus.cc:482] T 209ecdb78af240aeb4ae8a20046611ee P b499dd73acdb48028230a6e4f02bd2d4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 17:55:57.595803 16285 raft_consensus.cc:3037] T 209ecdb78af240aeb4ae8a20046611ee P b499dd73acdb48028230a6e4f02bd2d4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:55:57.596729 16285 raft_consensus.cc:504] T 209ecdb78af240aeb4ae8a20046611ee P b499dd73acdb48028230a6e4f02bd2d4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b499dd73acdb48028230a6e4f02bd2d4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33451 } } peers { permanent_uuid: "e8a734f480b34296ac1bb50572558f10" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34933 } } peers { permanent_uuid: "433a7f95a3694411b43c7a77d7fc93ea" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46395 } }
I20260929 17:55:57.597028 16285 leader_election.cc:288] T 209ecdb78af240aeb4ae8a20046611ee P b499dd73acdb48028230a6e4f02bd2d4 [CANDIDATE]: Term 1 election: Requested vote from peers e8a734f480b34296ac1bb50572558f10 (127.0.98.3:34933), 433a7f95a3694411b43c7a77d7fc93ea (127.0.98.1:46395)
I20260929 17:55:57.597318 16216 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "209ecdb78af240aeb4ae8a20046611ee" candidate_uuid: "b499dd73acdb48028230a6e4f02bd2d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8a734f480b34296ac1bb50572558f10"
I20260929 17:55:57.597330 15948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "209ecdb78af240aeb4ae8a20046611ee" candidate_uuid: "b499dd73acdb48028230a6e4f02bd2d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "433a7f95a3694411b43c7a77d7fc93ea"
I20260929 17:55:57.597412 16216 raft_consensus.cc:3037] T 209ecdb78af240aeb4ae8a20046611ee P e8a734f480b34296ac1bb50572558f10 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:55:57.597416 15948 raft_consensus.cc:3037] T 209ecdb78af240aeb4ae8a20046611ee P 433a7f95a3694411b43c7a77d7fc93ea [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:55:57.598341 15948 raft_consensus.cc:2454] T 209ecdb78af240aeb4ae8a20046611ee P 433a7f95a3694411b43c7a77d7fc93ea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b499dd73acdb48028230a6e4f02bd2d4 in term 1.
I20260929 17:55:57.598453 16216 raft_consensus.cc:2454] T 209ecdb78af240aeb4ae8a20046611ee P e8a734f480b34296ac1bb50572558f10 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b499dd73acdb48028230a6e4f02bd2d4 in term 1.
I20260929 17:55:57.598590 16016 leader_election.cc:302] T 209ecdb78af240aeb4ae8a20046611ee P b499dd73acdb48028230a6e4f02bd2d4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 433a7f95a3694411b43c7a77d7fc93ea, b499dd73acdb48028230a6e4f02bd2d4; no voters:
I20260929 17:55:57.598742 16285 raft_consensus.cc:2781] T 209ecdb78af240aeb4ae8a20046611ee P b499dd73acdb48028230a6e4f02bd2d4 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 17:55:57.598946 16285 raft_consensus.cc:686] T 209ecdb78af240aeb4ae8a20046611ee P b499dd73acdb48028230a6e4f02bd2d4 [term 1 LEADER]: Becoming Leader. State: Replica: b499dd73acdb48028230a6e4f02bd2d4, State: Running, Role: LEADER
I20260929 17:55:57.599259 16285 consensus_queue.cc:237] T 209ecdb78af240aeb4ae8a20046611ee P b499dd73acdb48028230a6e4f02bd2d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b499dd73acdb48028230a6e4f02bd2d4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33451 } } peers { permanent_uuid: "e8a734f480b34296ac1bb50572558f10" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34933 } } peers { permanent_uuid: "433a7f95a3694411b43c7a77d7fc93ea" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46395 } }
I20260929 17:55:57.602016 15800 catalog_manager.cc:5495] T 209ecdb78af240aeb4ae8a20046611ee P b499dd73acdb48028230a6e4f02bd2d4 reported cstate change: term changed from 0 to 1, leader changed from <none> to b499dd73acdb48028230a6e4f02bd2d4 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "b499dd73acdb48028230a6e4f02bd2d4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b499dd73acdb48028230a6e4f02bd2d4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33451 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e8a734f480b34296ac1bb50572558f10" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34933 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "433a7f95a3694411b43c7a77d7fc93ea" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46395 } health_report { overall_health: UNKNOWN } } }
I20260929 17:55:57.623782 16278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:55:57.620487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37466 (local address 127.0.98.2:33451)
0929 17:55:57.620607 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0929 17:55:57.620611 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:55:57.620650 (+ 39us) server_negotiation.cc:407] Connection header received
0929 17:55:57.620718 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:55:57.620720 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:55:57.620774 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:55:57.620873 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 17:55:57.621178 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:55:57.621664 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:55:57.622503 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:55:57.622699 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:55:57.622765 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:55:57.623038 (+ 273us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 17:55:57.623464 (+ 426us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 17:55:57.623551 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0929 17:55:57.623599 (+ 48us) server_negotiation.cc:299] Negotiation successful
0929 17:55:57.623649 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 17:55:57.629364 15948 raft_consensus.cc:1261] T 209ecdb78af240aeb4ae8a20046611ee P 433a7f95a3694411b43c7a77d7fc93ea [term 1 FOLLOWER]: Refusing update from remote peer b499dd73acdb48028230a6e4f02bd2d4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 17:55:57.629411 16216 raft_consensus.cc:1261] T 209ecdb78af240aeb4ae8a20046611ee P e8a734f480b34296ac1bb50572558f10 [term 1 FOLLOWER]: Refusing update from remote peer b499dd73acdb48028230a6e4f02bd2d4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 17:55:57.629987 16289 consensus_queue.cc:1035] T 209ecdb78af240aeb4ae8a20046611ee P b499dd73acdb48028230a6e4f02bd2d4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "433a7f95a3694411b43c7a77d7fc93ea" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 17:55:57.630162 16285 consensus_queue.cc:1035] T 209ecdb78af240aeb4ae8a20046611ee P b499dd73acdb48028230a6e4f02bd2d4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8a734f480b34296ac1bb50572558f10" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 17:55:57.637542 16293 mvcc.cc:205] Tried to move back new op lower bound from 7334725868043259904 to 7334725867928834048. Current Snapshot: MvccSnapshot[applied={T|T < 7334725868043259904}]
I20260929 17:55:57.637810 16294 mvcc.cc:205] Tried to move back new op lower bound from 7334725868043259904 to 7334725867928834048. Current Snapshot: MvccSnapshot[applied={T|T < 7334725868043259904}]
I20260929 17:55:57.645731 15800 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38312:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 17:55:57.645905 15800 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38312:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 17:55:57.648108 15800 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 842b44590ee242fca73034b1300da182: Sending DeleteTablet for 3 replicas of tablet 209ecdb78af240aeb4ae8a20046611ee
I20260929 17:55:57.648871 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskup3Soe/build/debug/bin/kudu with pid 15865
I20260929 17:55:57.648852 15928 tablet_service.cc:1514] Processing DeleteTablet for tablet 209ecdb78af240aeb4ae8a20046611ee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 17:55:57 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60886
I20260929 17:55:57.649091 16062 tablet_service.cc:1514] Processing DeleteTablet for tablet 209ecdb78af240aeb4ae8a20046611ee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 17:55:57 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37450
I20260929 17:55:57.649549 16196 tablet_service.cc:1514] Processing DeleteTablet for tablet 209ecdb78af240aeb4ae8a20046611ee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 17:55:57 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34608
I20260929 17:55:57.649972 16303 tablet_replica.cc:330] T 209ecdb78af240aeb4ae8a20046611ee P e8a734f480b34296ac1bb50572558f10: stopping tablet replica
I20260929 17:55:57.650278 16303 raft_consensus.cc:2229] T 209ecdb78af240aeb4ae8a20046611ee P e8a734f480b34296ac1bb50572558f10 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 17:55:57.650605 16303 raft_consensus.cc:2258] T 209ecdb78af240aeb4ae8a20046611ee P e8a734f480b34296ac1bb50572558f10 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 17:55:57.651880 16303 ts_tablet_manager.cc:1905] T 209ecdb78af240aeb4ae8a20046611ee P e8a734f480b34296ac1bb50572558f10: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 17:55:57.653951 16303 ts_tablet_manager.cc:1918] T 209ecdb78af240aeb4ae8a20046611ee P e8a734f480b34296ac1bb50572558f10: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 17:55:57.654030 16303 log.cc:1196] T 209ecdb78af240aeb4ae8a20046611ee P e8a734f480b34296ac1bb50572558f10: Deleting WAL directory at /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/minicluster-data/ts-2/wal/wals/209ecdb78af240aeb4ae8a20046611ee
I20260929 17:55:57.654280 16303 ts_tablet_manager.cc:1939] T 209ecdb78af240aeb4ae8a20046611ee P e8a734f480b34296ac1bb50572558f10: Deleting consensus metadata
I20260929 17:55:57.655311 15787 catalog_manager.cc:4841] TS e8a734f480b34296ac1bb50572558f10 (127.0.98.3:34933): tablet 209ecdb78af240aeb4ae8a20046611ee (table test-table [id=8cbc0e9f803d411a9f7c70253d10ba55]) successfully deleted
W20260929 17:55:57.656622 15788 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:46395 (error 108)
I20260929 17:55:57.657395 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskup3Soe/build/debug/bin/kudu with pid 15999
I20260929 17:55:57.657462 16302 tablet_replica.cc:330] T 209ecdb78af240aeb4ae8a20046611ee P b499dd73acdb48028230a6e4f02bd2d4: stopping tablet replica
I20260929 17:55:57.657653 16274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:55:57.657368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46395 (local address 127.0.0.1:60890)
0929 17:55:57.657523 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0929 17:55:57.657579 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260929 17:55:57.657744 16302 raft_consensus.cc:2229] T 209ecdb78af240aeb4ae8a20046611ee P b499dd73acdb48028230a6e4f02bd2d4 [term 1 LEADER]: Raft consensus shutting down.
W20260929 17:55:57.657778 15788 catalog_manager.cc:4584] TS 433a7f95a3694411b43c7a77d7fc93ea (127.0.98.1:46395): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 209ecdb78af240aeb4ae8a20046611ee: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46395: connect: Connection refused (error 111)
I20260929 17:55:57.662667 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskup3Soe/build/debug/bin/kudu with pid 16133
I20260929 17:55:57.663632 16274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:55:57.663346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33451 (local address 127.0.0.1:37474)
0929 17:55:57.663509 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0929 17:55:57.663558 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:33451: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260929 17:55:57.667219 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskup3Soe/build/debug/bin/kudu with pid 15771
I20260929 17:56:17.672314 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskup3Soe/build/debug/bin/kudu
/tmp/dist-test-taskup3Soe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33709
--webserver_interface=127.0.98.62
--webserver_port=43621
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33709
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:56:17.779008 16305 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:56:17.779309 16305 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:56:17.779373 16305 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:56:17.783435 16305 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260929 17:56:17.783494 16305 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260929 17:56:17.783516 16305 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260929 17:56:17.783533 16305 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 17:56:17.783557 16305 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:56:17.783591 16305 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:56:17.783618 16305 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 17:56:17.783648 16305 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 17:56:17.788918 16305 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33709
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33709
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=43621
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16305
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:56:17.790009 16305 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:56:17.791028 16305 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:56:17.803875 16312 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:56:17.803804 16311 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:56:17.803891 16314 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:56:17.803828 16305 server_base.cc:1034] running on GCE node
I20260929 17:56:17.804718 16305 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:56:17.805910 16305 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:56:17.807152 16305 hybrid_clock.cc:648] HybridClock initialized: now 1790704577807118 us; error 43 us; skew 500 ppm
Sep 29 17:56:17 dist-test-slave-53nh krb5kdc[15758](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704577, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:56:17.815156 16305 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 17:56:17.816172 16305 webserver.cc:460] Webserver started at http://127.0.98.62:43621/ using document root <none> and password file <none>
I20260929 17:56:17.816723 16305 fs_manager.cc:359] Metadata directory not provided
I20260929 17:56:17.816792 16305 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:56:17.820655 16305 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260929 17:56:17.822788 16321 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:56:17.823980 16305 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260929 17:56:17.824079 16305 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/minicluster-data/master-0/wal
uuid: "842b44590ee242fca73034b1300da182"
format_stamp: "Formatted at 2026-09-29 17:55:56 on dist-test-slave-53nh"
server_key: "8021b0090bb6c1c80b9b0f788960a2e6"
server_key_iv: "f505f861ff6bb656c97c8657b4915be4"
server_key_version: "encryptionkey@0"
I20260929 17:56:17.824636 16305 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:56:17.852070 16305 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:56:17.854956 16305 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:56:17.855108 16305 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:56:17.861752 16305 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33709
I20260929 17:56:17.862018 16373 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33709 every 8 connection(s)
I20260929 17:56:17.862833 16305 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/minicluster-data/master-0/data/info.pb
I20260929 17:56:17.867148 16374 sys_catalog.cc:263] Verifying existing consensus state
I20260929 17:56:17.868772 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskup3Soe/build/debug/bin/kudu as pid 16305
I20260929 17:56:17.869195 16374 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 842b44590ee242fca73034b1300da182: Bootstrap starting.
I20260929 17:56:17.869279 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskup3Soe/build/debug/bin/kudu
/tmp/dist-test-taskup3Soe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:46395
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:33709
--webserver_port=43617
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 17:56:17.886770 16374 log.cc:824] T 00000000000000000000000000000000 P 842b44590ee242fca73034b1300da182: Log is configured to *not* fsync() on all Append() calls
I20260929 17:56:17.895480 16374 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 842b44590ee242fca73034b1300da182: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 17:56:17.895853 16374 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 842b44590ee242fca73034b1300da182: Bootstrap complete.
I20260929 17:56:17.899818 16374 raft_consensus.cc:348] T 00000000000000000000000000000000 P 842b44590ee242fca73034b1300da182 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "842b44590ee242fca73034b1300da182" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33709 } }
I20260929 17:56:17.900215 16374 raft_consensus.cc:729] T 00000000000000000000000000000000 P 842b44590ee242fca73034b1300da182 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 842b44590ee242fca73034b1300da182, State: Initialized, Role: FOLLOWER
I20260929 17:56:17.900662 16374 consensus_queue.cc:260] T 00000000000000000000000000000000 P 842b44590ee242fca73034b1300da182 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "842b44590ee242fca73034b1300da182" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33709 } }
I20260929 17:56:17.900782 16374 raft_consensus.cc:388] T 00000000000000000000000000000000 P 842b44590ee242fca73034b1300da182 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 17:56:17.900849 16374 raft_consensus.cc:482] T 00000000000000000000000000000000 P 842b44590ee242fca73034b1300da182 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 17:56:17.900938 16374 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 842b44590ee242fca73034b1300da182 [term 1 FOLLOWER]: Advancing to term 2
I20260929 17:56:17.902834 16374 raft_consensus.cc:504] T 00000000000000000000000000000000 P 842b44590ee242fca73034b1300da182 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "842b44590ee242fca73034b1300da182" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33709 } }
I20260929 17:56:17.903167 16374 leader_election.cc:302] T 00000000000000000000000000000000 P 842b44590ee242fca73034b1300da182 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 842b44590ee242fca73034b1300da182; no voters:
I20260929 17:56:17.903481 16374 leader_election.cc:288] T 00000000000000000000000000000000 P 842b44590ee242fca73034b1300da182 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260929 17:56:17.903584 16378 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 842b44590ee242fca73034b1300da182 [term 2 FOLLOWER]: Leader election won for term 2
I20260929 17:56:17.903786 16378 raft_consensus.cc:686] T 00000000000000000000000000000000 P 842b44590ee242fca73034b1300da182 [term 2 LEADER]: Becoming Leader. State: Replica: 842b44590ee242fca73034b1300da182, State: Running, Role: LEADER
I20260929 17:56:17.904137 16378 consensus_queue.cc:237] T 00000000000000000000000000000000 P 842b44590ee242fca73034b1300da182 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "842b44590ee242fca73034b1300da182" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33709 } }
I20260929 17:56:17.904707 16374 sys_catalog.cc:564] T 00000000000000000000000000000000 P 842b44590ee242fca73034b1300da182 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 17:56:17.905094 16378 sys_catalog.cc:455] T 00000000000000000000000000000000 P 842b44590ee242fca73034b1300da182 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 842b44590ee242fca73034b1300da182. Latest consensus state: current_term: 2 leader_uuid: "842b44590ee242fca73034b1300da182" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "842b44590ee242fca73034b1300da182" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33709 } } }
I20260929 17:56:17.905215 16378 sys_catalog.cc:458] T 00000000000000000000000000000000 P 842b44590ee242fca73034b1300da182 [sys.catalog]: This master's current role is: LEADER
I20260929 17:56:17.905582 16379 sys_catalog.cc:455] T 00000000000000000000000000000000 P 842b44590ee242fca73034b1300da182 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "842b44590ee242fca73034b1300da182" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "842b44590ee242fca73034b1300da182" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33709 } } }
I20260929 17:56:17.905745 16379 sys_catalog.cc:458] T 00000000000000000000000000000000 P 842b44590ee242fca73034b1300da182 [sys.catalog]: This master's current role is: LEADER
I20260929 17:56:17.906037 16382 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 17:56:17.910766 16382 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 17:56:17.911442 16382 catalog_manager.cc:1259] Loaded cluster ID: 92f56c901b26449fbe079d5de709dbdd
I20260929 17:56:17.911500 16382 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 17:56:17.914050 16382 catalog_manager.cc:1504] Loading token signing keys...
I20260929 17:56:17.914722 16382 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 842b44590ee242fca73034b1300da182: Loaded TSK: 0
I20260929 17:56:17.932418 16382 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 842b44590ee242fca73034b1300da182: Generated new TSK 1
I20260929 17:56:17.932710 16382 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 17:56:17.985682 16376 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:56:17.985946 16376 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:56:17.985989 16376 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:56:17.989979 16376 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:56:17.990043 16376 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:56:17.990104 16376 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 17:56:17.995316 16376 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:46395
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=43617
--tserver_master_addrs=127.0.98.62:33709
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16376
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:56:17.996381 16376 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:56:17.997481 16376 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:56:18.008280 16404 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:56:18.008388 16402 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:56:18.008280 16401 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:56:18.008649 16376 server_base.cc:1034] running on GCE node
I20260929 17:56:18.009107 16376 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:56:18.009759 16376 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:56:18.010938 16376 hybrid_clock.cc:648] HybridClock initialized: now 1790704578010903 us; error 54 us; skew 500 ppm
Sep 29 17:56:18 dist-test-slave-53nh krb5kdc[15758](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704578, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:56:18.013641 16376 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260929 17:56:18.014519 16376 webserver.cc:460] Webserver started at http://127.0.98.1:43617/ using document root <none> and password file <none>
I20260929 17:56:18.015115 16376 fs_manager.cc:359] Metadata directory not provided
I20260929 17:56:18.015197 16376 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:56:18.019361 16376 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 17:56:18.021644 16411 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:56:18.022924 16376 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 17:56:18.023029 16376 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/minicluster-data/ts-0/wal
uuid: "433a7f95a3694411b43c7a77d7fc93ea"
format_stamp: "Formatted at 2026-09-29 17:55:57 on dist-test-slave-53nh"
server_key: "232988f4455d1b6900d092f246158ce9"
server_key_iv: "c2e13854c1871df45f647fde8765fdfa"
server_key_version: "encryptionkey@0"
I20260929 17:56:18.023578 16376 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:56:18.037787 16376 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:56:18.040926 16376 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:56:18.041128 16376 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:56:18.041716 16376 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 17:56:18.042802 16418 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260929 17:56:18.046748 16376 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260929 17:56:18.046801 16376 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20260929 17:56:18.046849 16376 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260929 17:56:18.048441 16376 ts_tablet_manager.cc:610] Registered 1 tablets
I20260929 17:56:18.048496 16376 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260929 17:56:18.048770 16418 tablet_bootstrap.cc:492] T 209ecdb78af240aeb4ae8a20046611ee P 433a7f95a3694411b43c7a77d7fc93ea: Bootstrap starting.
I20260929 17:56:18.058826 16376 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46395
I20260929 17:56:18.058846 16525 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46395 every 8 connection(s)
I20260929 17:56:18.060086 16376 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/minicluster-data/ts-0/data/info.pb
I20260929 17:56:18.065675 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskup3Soe/build/debug/bin/kudu as pid 16376
I20260929 17:56:18.066072 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskup3Soe/build/debug/bin/kudu
/tmp/dist-test-taskup3Soe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:33451
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:33709
--webserver_port=35103
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 17:56:18.068877 16418 log.cc:824] T 209ecdb78af240aeb4ae8a20046611ee P 433a7f95a3694411b43c7a77d7fc93ea: Log is configured to *not* fsync() on all Append() calls
Sep 29 17:56:18 dist-test-slave-53nh krb5kdc[15758](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704578, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 17:56:18.076606 16418 tablet_bootstrap.cc:492] T 209ecdb78af240aeb4ae8a20046611ee P 433a7f95a3694411b43c7a77d7fc93ea: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 17:56:18.076959 16418 tablet_bootstrap.cc:492] T 209ecdb78af240aeb4ae8a20046611ee P 433a7f95a3694411b43c7a77d7fc93ea: Bootstrap complete.
I20260929 17:56:18.077895 16418 ts_tablet_manager.cc:1397] T 209ecdb78af240aeb4ae8a20046611ee P 433a7f95a3694411b43c7a77d7fc93ea: Time spent bootstrapping tablet: real 0.029s user 0.021s sys 0.007s
I20260929 17:56:18.078572 16528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:56:18.062080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33709 (local address 127.0.98.1:54699)
0929 17:56:18.062665 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0929 17:56:18.062743 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0929 17:56:18.063834 (+ 1091us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:56:18.065049 (+ 1215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:56:18.065058 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:56:18.065619 (+ 561us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:56:18.066252 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:56:18.066271 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:56:18.067941 (+ 1670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:56:18.067943 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:56:18.068801 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:56:18.068812 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:56:18.069657 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:56:18.069661 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:56:18.069848 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:56:18.070526 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:56:18.070547 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:56:18.072376 (+ 1829us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:56:18.074964 (+ 2588us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:56:18.074971 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:56:18.074985 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:56:18.075332 (+ 347us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:56:18.075792 (+ 460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:56:18.075795 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:56:18.075797 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:56:18.075966 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:56:18.076747 (+ 781us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:56:18.076753 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:56:18.077109 (+ 356us) client_negotiation.cc:769] Sending connection context
0929 17:56:18.077299 (+ 190us) client_negotiation.cc:240] Negotiation successful
0929 17:56:18.077636 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":122,"threads_started":1}
I20260929 17:56:18.080379 16529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:56:18.062138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54699 (local address 127.0.98.62:33709)
0929 17:56:18.062639 (+ 501us) server_negotiation.cc:206] Beginning negotiation
0929 17:56:18.062657 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 17:56:18.062871 (+ 214us) server_negotiation.cc:407] Connection header received
0929 17:56:18.064288 (+ 1417us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:56:18.064300 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:56:18.064598 (+ 298us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:56:18.064869 (+ 271us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:56:18.066440 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:56:18.067790 (+ 1350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:56:18.068921 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:56:18.069513 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:56:18.069683 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:56:18.072613 (+ 2930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:56:18.072620 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:56:18.072639 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:56:18.072674 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:56:18.074820 (+ 2146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:56:18.075496 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:56:18.075502 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:56:18.075508 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:56:18.075672 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:56:18.076067 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:56:18.076070 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:56:18.076073 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:56:18.076600 (+ 527us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:56:18.079046 (+ 2446us) server_negotiation.cc:1035] Waiting for connection context
0929 17:56:18.079194 (+ 148us) server_negotiation.cc:299] Negotiation successful
0929 17:56:18.079508 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":377,"thread_start_us":142,"threads_started":1}
I20260929 17:56:18.080583 16418 raft_consensus.cc:348] T 209ecdb78af240aeb4ae8a20046611ee P 433a7f95a3694411b43c7a77d7fc93ea [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b499dd73acdb48028230a6e4f02bd2d4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33451 } } peers { permanent_uuid: "e8a734f480b34296ac1bb50572558f10" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34933 } } peers { permanent_uuid: "433a7f95a3694411b43c7a77d7fc93ea" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46395 } }
I20260929 17:56:18.081125 16418 raft_consensus.cc:729] T 209ecdb78af240aeb4ae8a20046611ee P 433a7f95a3694411b43c7a77d7fc93ea [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 433a7f95a3694411b43c7a77d7fc93ea, State: Initialized, Role: FOLLOWER
I20260929 17:56:18.081593 16418 consensus_queue.cc:260] T 209ecdb78af240aeb4ae8a20046611ee P 433a7f95a3694411b43c7a77d7fc93ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b499dd73acdb48028230a6e4f02bd2d4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33451 } } peers { permanent_uuid: "e8a734f480b34296ac1bb50572558f10" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34933 } } peers { permanent_uuid: "433a7f95a3694411b43c7a77d7fc93ea" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46395 } }
I20260929 17:56:18.082581 16418 ts_tablet_manager.cc:1428] T 209ecdb78af240aeb4ae8a20046611ee P 433a7f95a3694411b43c7a77d7fc93ea: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260929 17:56:18.083921 16526 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33709
I20260929 17:56:18.084164 16526 heartbeater.cc:464] Registering TS with master...
I20260929 17:56:18.084749 16526 heartbeater.cc:511] Master 127.0.98.62:33709 requested a full tablet report, sending...
I20260929 17:56:18.087536 16338 ts_manager.cc:194] Registered new tserver with Master: 433a7f95a3694411b43c7a77d7fc93ea (127.0.98.1:46395)
I20260929 17:56:18.091634 16338 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:54699
I20260929 17:56:18.094187 16526 heartbeater.cc:503] Master 127.0.98.62:33709 was elected leader, sending a full tablet report...
Sep 29 17:56:18 dist-test-slave-53nh krb5kdc[15758](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704577, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260929 17:56:18.101490 16533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:56:18.091067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46395 (local address 127.0.0.1:55444)
0929 17:56:18.091817 (+ 750us) negotiation.cc:107] Waiting for socket to connect
0929 17:56:18.091864 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 17:56:18.092009 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:56:18.092558 (+ 549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:56:18.092563 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:56:18.092980 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:56:18.093152 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:56:18.093160 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:56:18.094488 (+ 1328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:56:18.094492 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:56:18.095120 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:56:18.095129 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:56:18.095767 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:56:18.095784 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:56:18.095895 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:56:18.096404 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:56:18.096416 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:56:18.097811 (+ 1395us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:56:18.099786 (+ 1975us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:56:18.099795 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:56:18.099800 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:56:18.100011 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:56:18.100333 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:56:18.100336 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:56:18.100337 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:56:18.100431 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:56:18.100918 (+ 487us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:56:18.100922 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:56:18.101072 (+ 150us) client_negotiation.cc:769] Sending connection context
0929 17:56:18.101189 (+ 117us) client_negotiation.cc:240] Negotiation successful
0929 17:56:18.101359 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":579,"thread_start_us":126,"threads_started":1}
I20260929 17:56:18.101613 16534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:56:18.091137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55444 (local address 127.0.98.1:46395)
0929 17:56:18.091564 (+ 427us) server_negotiation.cc:206] Beginning negotiation
0929 17:56:18.091575 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 17:56:18.091959 (+ 384us) server_negotiation.cc:407] Connection header received
0929 17:56:18.092149 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:56:18.092153 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:56:18.092310 (+ 157us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:56:18.092423 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:56:18.093272 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:56:18.094352 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:56:18.095231 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:56:18.095550 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:56:18.095643 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:56:18.097946 (+ 2303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:56:18.097957 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:56:18.097969 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:56:18.097999 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:56:18.099678 (+ 1679us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:56:18.100114 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:56:18.100121 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:56:18.100123 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:56:18.100216 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:56:18.100515 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:56:18.100518 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:56:18.100519 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:56:18.100806 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:56:18.100932 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0929 17:56:18.101279 (+ 347us) server_negotiation.cc:299] Negotiation successful
0929 17:56:18.101455 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":108,"threads_started":1}
I20260929 17:56:18.102557 16460 tablet_service.cc:1514] Processing DeleteTablet for tablet 209ecdb78af240aeb4ae8a20046611ee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 17:55:57 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55444
I20260929 17:56:18.103566 16535 tablet_replica.cc:330] stopping tablet replica
I20260929 17:56:18.103833 16535 raft_consensus.cc:2229] T 209ecdb78af240aeb4ae8a20046611ee P 433a7f95a3694411b43c7a77d7fc93ea [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 17:56:18.104064 16535 raft_consensus.cc:2258] T 209ecdb78af240aeb4ae8a20046611ee P 433a7f95a3694411b43c7a77d7fc93ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 17:56:18.104976 16535 ts_tablet_manager.cc:1905] T 209ecdb78af240aeb4ae8a20046611ee P 433a7f95a3694411b43c7a77d7fc93ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 17:56:18.108711 16535 ts_tablet_manager.cc:1918] T 209ecdb78af240aeb4ae8a20046611ee P 433a7f95a3694411b43c7a77d7fc93ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 17:56:18.108803 16535 log.cc:1196] T 209ecdb78af240aeb4ae8a20046611ee P 433a7f95a3694411b43c7a77d7fc93ea: Deleting WAL directory at /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/minicluster-data/ts-0/wal/wals/209ecdb78af240aeb4ae8a20046611ee
I20260929 17:56:18.109059 16535 ts_tablet_manager.cc:1939] T 209ecdb78af240aeb4ae8a20046611ee P 433a7f95a3694411b43c7a77d7fc93ea: Deleting consensus metadata
I20260929 17:56:18.110874 16323 catalog_manager.cc:4841] TS 433a7f95a3694411b43c7a77d7fc93ea (127.0.98.1:46395): tablet 209ecdb78af240aeb4ae8a20046611ee (table test-table [id=8cbc0e9f803d411a9f7c70253d10ba55]) successfully deleted
W20260929 17:56:18.183149 16530 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:56:18.183414 16530 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:56:18.183459 16530 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:56:18.187338 16530 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:56:18.187386 16530 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:56:18.187441 16530 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 17:56:18.192457 16530 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:33451
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=35103
--tserver_master_addrs=127.0.98.62:33709
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16530
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:56:18.193439 16530 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:56:18.194375 16530 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:56:18.203416 16540 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:56:18.203580 16530 server_base.cc:1034] running on GCE node
W20260929 17:56:18.203454 16541 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:56:18.203440 16543 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:56:18.204150 16530 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:56:18.204769 16530 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:56:18.205915 16530 hybrid_clock.cc:648] HybridClock initialized: now 1790704578205887 us; error 39 us; skew 500 ppm
Sep 29 17:56:18 dist-test-slave-53nh krb5kdc[15758](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704578, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:56:18.208613 16530 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260929 17:56:18.209473 16530 webserver.cc:460] Webserver started at http://127.0.98.2:35103/ using document root <none> and password file <none>
I20260929 17:56:18.209969 16530 fs_manager.cc:359] Metadata directory not provided
I20260929 17:56:18.210032 16530 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:56:18.213738 16530 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260929 17:56:18.215739 16550 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:56:18.216938 16530 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 17:56:18.217068 16530 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/minicluster-data/ts-1/wal
uuid: "b499dd73acdb48028230a6e4f02bd2d4"
format_stamp: "Formatted at 2026-09-29 17:55:57 on dist-test-slave-53nh"
server_key: "d2473c82904daeddcebaac9867c45400"
server_key_iv: "1943549757123a78e76af5f53abfd36d"
server_key_version: "encryptionkey@0"
I20260929 17:56:18.217581 16530 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:56:18.229764 16530 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:56:18.232898 16530 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:56:18.233078 16530 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:56:18.233667 16530 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 17:56:18.234676 16557 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260929 17:56:18.238833 16530 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260929 17:56:18.238903 16530 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20260929 17:56:18.238969 16530 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260929 17:56:18.240738 16530 ts_tablet_manager.cc:610] Registered 1 tablets
I20260929 17:56:18.240785 16530 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260929 17:56:18.241075 16557 tablet_bootstrap.cc:492] T 209ecdb78af240aeb4ae8a20046611ee P b499dd73acdb48028230a6e4f02bd2d4: Bootstrap starting.
I20260929 17:56:18.250209 16530 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33451
I20260929 17:56:18.250281 16664 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33451 every 8 connection(s)
I20260929 17:56:18.251164 16530 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/minicluster-data/ts-1/data/info.pb
I20260929 17:56:18.253199 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskup3Soe/build/debug/bin/kudu as pid 16530
I20260929 17:56:18.253666 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskup3Soe/build/debug/bin/kudu
/tmp/dist-test-taskup3Soe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:34933
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:33709
--webserver_port=46089
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 17:56:18.258565 16557 log.cc:824] T 209ecdb78af240aeb4ae8a20046611ee P b499dd73acdb48028230a6e4f02bd2d4: Log is configured to *not* fsync() on all Append() calls
Sep 29 17:56:18 dist-test-slave-53nh krb5kdc[15758](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704578, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 17:56:18.265199 16529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:56:18.252964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54355 (local address 127.0.98.62:33709)
0929 17:56:18.253157 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0929 17:56:18.253162 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:56:18.253732 (+ 570us) server_negotiation.cc:407] Connection header received
0929 17:56:18.254829 (+ 1097us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:56:18.254833 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:56:18.254888 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:56:18.254962 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:56:18.256450 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:56:18.256987 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:56:18.258130 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:56:18.258359 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:56:18.258448 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:56:18.261206 (+ 2758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:56:18.261211 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:56:18.261214 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:56:18.261248 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:56:18.262940 (+ 1692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:56:18.263529 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:56:18.263533 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:56:18.263535 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:56:18.263627 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:56:18.263980 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:56:18.263983 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:56:18.263985 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:56:18.264215 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:56:18.264320 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0929 17:56:18.264859 (+ 539us) server_negotiation.cc:299] Negotiation successful
0929 17:56:18.265028 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260929 17:56:18.266022 16667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:56:18.253085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33709 (local address 127.0.98.2:54355)
0929 17:56:18.253580 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0929 17:56:18.253630 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0929 17:56:18.254630 (+ 1000us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:56:18.255130 (+ 500us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:56:18.255140 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:56:18.255629 (+ 489us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:56:18.256299 (+ 670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:56:18.256319 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:56:18.257149 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:56:18.257152 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:56:18.258015 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:56:18.258024 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:56:18.258474 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:56:18.258478 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:56:18.258654 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:56:18.259255 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:56:18.259274 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:56:18.261021 (+ 1747us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:56:18.263063 (+ 2042us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:56:18.263068 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:56:18.263081 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:56:18.263427 (+ 346us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:56:18.263721 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:56:18.263724 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:56:18.263727 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:56:18.263888 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:56:18.264324 (+ 436us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:56:18.264330 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:56:18.264606 (+ 276us) client_negotiation.cc:769] Sending connection context
0929 17:56:18.264795 (+ 189us) client_negotiation.cc:240] Negotiation successful
0929 17:56:18.265125 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":124,"threads_started":1}
I20260929 17:56:18.266451 16557 tablet_bootstrap.cc:492] T 209ecdb78af240aeb4ae8a20046611ee P b499dd73acdb48028230a6e4f02bd2d4: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 17:56:18.266855 16557 tablet_bootstrap.cc:492] T 209ecdb78af240aeb4ae8a20046611ee P b499dd73acdb48028230a6e4f02bd2d4: Bootstrap complete.
I20260929 17:56:18.267216 16665 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33709
I20260929 17:56:18.267459 16665 heartbeater.cc:464] Registering TS with master...
I20260929 17:56:18.267820 16557 ts_tablet_manager.cc:1397] T 209ecdb78af240aeb4ae8a20046611ee P b499dd73acdb48028230a6e4f02bd2d4: Time spent bootstrapping tablet: real 0.027s user 0.016s sys 0.007s
I20260929 17:56:18.267977 16665 heartbeater.cc:511] Master 127.0.98.62:33709 requested a full tablet report, sending...
I20260929 17:56:18.270040 16338 ts_manager.cc:194] Registered new tserver with Master: b499dd73acdb48028230a6e4f02bd2d4 (127.0.98.2:33451)
I20260929 17:56:18.270480 16557 raft_consensus.cc:348] T 209ecdb78af240aeb4ae8a20046611ee P b499dd73acdb48028230a6e4f02bd2d4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b499dd73acdb48028230a6e4f02bd2d4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33451 } } peers { permanent_uuid: "e8a734f480b34296ac1bb50572558f10" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34933 } } peers { permanent_uuid: "433a7f95a3694411b43c7a77d7fc93ea" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46395 } }
I20260929 17:56:18.270952 16557 raft_consensus.cc:729] T 209ecdb78af240aeb4ae8a20046611ee P b499dd73acdb48028230a6e4f02bd2d4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b499dd73acdb48028230a6e4f02bd2d4, State: Initialized, Role: FOLLOWER
I20260929 17:56:18.271368 16338 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:54355
I20260929 17:56:18.271484 16557 consensus_queue.cc:260] T 209ecdb78af240aeb4ae8a20046611ee P b499dd73acdb48028230a6e4f02bd2d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b499dd73acdb48028230a6e4f02bd2d4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33451 } } peers { permanent_uuid: "e8a734f480b34296ac1bb50572558f10" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34933 } } peers { permanent_uuid: "433a7f95a3694411b43c7a77d7fc93ea" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46395 } }
I20260929 17:56:18.272485 16557 ts_tablet_manager.cc:1428] T 209ecdb78af240aeb4ae8a20046611ee P b499dd73acdb48028230a6e4f02bd2d4: Time spent starting tablet: real 0.005s user 0.007s sys 0.000s
I20260929 17:56:18.273726 16665 heartbeater.cc:503] Master 127.0.98.62:33709 was elected leader, sending a full tablet report...
I20260929 17:56:18.277510 16533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:56:18.271048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33451 (local address 127.0.0.1:37164)
0929 17:56:18.271249 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0929 17:56:18.271265 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 17:56:18.271348 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:56:18.273746 (+ 2398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:56:18.273749 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:56:18.273770 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:56:18.273870 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:56:18.273877 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:56:18.275254 (+ 1377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:56:18.275256 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:56:18.276052 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:56:18.276062 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:56:18.277108 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:56:18.277111 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:56:18.277214 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:56:18.277261 (+ 47us) client_negotiation.cc:769] Sending connection context
0929 17:56:18.277323 (+ 62us) client_negotiation.cc:240] Negotiation successful
0929 17:56:18.277401 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115}
I20260929 17:56:18.277587 16671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:56:18.271701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37164 (local address 127.0.98.2:33451)
0929 17:56:18.273224 (+ 1523us) server_negotiation.cc:206] Beginning negotiation
0929 17:56:18.273236 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 17:56:18.273260 (+ 24us) server_negotiation.cc:407] Connection header received
0929 17:56:18.273329 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:56:18.273334 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:56:18.273563 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:56:18.273717 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:56:18.273983 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:56:18.275138 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:56:18.276170 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:56:18.276969 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:56:18.277105 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:56:18.277242 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0929 17:56:18.277370 (+ 128us) server_negotiation.cc:299] Negotiation successful
0929 17:56:18.277459 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1418,"thread_start_us":105,"threads_started":1}
I20260929 17:56:18.278483 16599 tablet_service.cc:1514] Processing DeleteTablet for tablet 209ecdb78af240aeb4ae8a20046611ee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 17:55:57 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37164
I20260929 17:56:18.279320 16672 tablet_replica.cc:330] stopping tablet replica
I20260929 17:56:18.279640 16672 raft_consensus.cc:2229] T 209ecdb78af240aeb4ae8a20046611ee P b499dd73acdb48028230a6e4f02bd2d4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 17:56:18.279891 16672 raft_consensus.cc:2258] T 209ecdb78af240aeb4ae8a20046611ee P b499dd73acdb48028230a6e4f02bd2d4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 17:56:18.280963 16672 ts_tablet_manager.cc:1905] T 209ecdb78af240aeb4ae8a20046611ee P b499dd73acdb48028230a6e4f02bd2d4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 17:56:18.285377 16672 ts_tablet_manager.cc:1918] T 209ecdb78af240aeb4ae8a20046611ee P b499dd73acdb48028230a6e4f02bd2d4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 17:56:18.285475 16672 log.cc:1196] T 209ecdb78af240aeb4ae8a20046611ee P b499dd73acdb48028230a6e4f02bd2d4: Deleting WAL directory at /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/minicluster-data/ts-1/wal/wals/209ecdb78af240aeb4ae8a20046611ee
I20260929 17:56:18.285725 16672 ts_tablet_manager.cc:1939] T 209ecdb78af240aeb4ae8a20046611ee P b499dd73acdb48028230a6e4f02bd2d4: Deleting consensus metadata
I20260929 17:56:18.287545 16322 catalog_manager.cc:4841] TS b499dd73acdb48028230a6e4f02bd2d4 (127.0.98.2:33451): tablet 209ecdb78af240aeb4ae8a20046611ee (table test-table [id=8cbc0e9f803d411a9f7c70253d10ba55]) successfully deleted
W20260929 17:56:18.370779 16668 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:56:18.371017 16668 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:56:18.371057 16668 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:56:18.374913 16668 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:56:18.374963 16668 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:56:18.375018 16668 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 17:56:18.380030 16668 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:34933
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=46089
--tserver_master_addrs=127.0.98.62:33709
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16668
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:56:18.380955 16668 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:56:18.381922 16668 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:56:18.390836 16678 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:56:18.390919 16668 server_base.cc:1034] running on GCE node
W20260929 17:56:18.390882 16677 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:56:18.390836 16680 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:56:18.391436 16668 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:56:18.392113 16668 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:56:18.393271 16668 hybrid_clock.cc:648] HybridClock initialized: now 1790704578393247 us; error 38 us; skew 500 ppm
Sep 29 17:56:18 dist-test-slave-53nh krb5kdc[15758](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704578, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:56:18.395974 16668 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260929 17:56:18.396787 16668 webserver.cc:460] Webserver started at http://127.0.98.3:46089/ using document root <none> and password file <none>
I20260929 17:56:18.397358 16668 fs_manager.cc:359] Metadata directory not provided
I20260929 17:56:18.397428 16668 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:56:18.401214 16668 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 17:56:18.403319 16687 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:56:18.404569 16668 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 17:56:18.404677 16668 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/minicluster-data/ts-2/wal
uuid: "e8a734f480b34296ac1bb50572558f10"
format_stamp: "Formatted at 2026-09-29 17:55:57 on dist-test-slave-53nh"
server_key: "43d0aeb3bdff69a5cbc3dca0926f9313"
server_key_iv: "d6083adc1f9f419fab5de7ef217989b3"
server_key_version: "encryptionkey@0"
I20260929 17:56:18.405249 16668 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:56:18.417672 16668 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:56:18.420840 16668 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:56:18.421067 16668 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:56:18.421700 16668 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 17:56:18.422577 16668 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:56:18.422636 16668 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:56:18.422696 16668 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:56:18.422739 16668 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 17:56:18.432108 16668 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34933
I20260929 17:56:18.432150 16800 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34933 every 8 connection(s)
I20260929 17:56:18.433266 16668 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/minicluster-data/ts-2/data/info.pb
Sep 29 17:56:18 dist-test-slave-53nh krb5kdc[15758](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704578, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 17:56:18.442977 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskup3Soe/build/debug/bin/kudu as pid 16668
I20260929 17:56:18.446273 16529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:56:18.435108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49559 (local address 127.0.98.62:33709)
0929 17:56:18.435253 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0929 17:56:18.435257 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:56:18.436047 (+ 790us) server_negotiation.cc:407] Connection header received
0929 17:56:18.437368 (+ 1321us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:56:18.437372 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:56:18.437427 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:56:18.437504 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:56:18.438656 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:56:18.439180 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:56:18.440282 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:56:18.440508 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:56:18.440598 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:56:18.442955 (+ 2357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:56:18.442958 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:56:18.442960 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:56:18.442985 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:56:18.444388 (+ 1403us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:56:18.444842 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:56:18.444845 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:56:18.444847 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:56:18.444921 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:56:18.445250 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:56:18.445253 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:56:18.445255 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:56:18.445438 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:56:18.445519 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0929 17:56:18.446001 (+ 482us) server_negotiation.cc:299] Negotiation successful
0929 17:56:18.446115 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 17:56:18.447036 16803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:56:18.435476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33709 (local address 127.0.98.3:49559)
0929 17:56:18.435908 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0929 17:56:18.435947 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 17:56:18.437151 (+ 1204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:56:18.437686 (+ 535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:56:18.437693 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:56:18.438046 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:56:18.438517 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:56:18.438530 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:56:18.439299 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:56:18.439302 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:56:18.440160 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:56:18.440169 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:56:18.440647 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:56:18.440652 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:56:18.440823 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:56:18.441398 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:56:18.441414 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:56:18.442789 (+ 1375us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:56:18.444497 (+ 1708us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:56:18.444502 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:56:18.444516 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:56:18.444750 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:56:18.445029 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:56:18.445032 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:56:18.445034 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:56:18.445161 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:56:18.445531 (+ 370us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:56:18.445538 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:56:18.445760 (+ 222us) client_negotiation.cc:769] Sending connection context
0929 17:56:18.445925 (+ 165us) client_negotiation.cc:240] Negotiation successful
0929 17:56:18.446237 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":114,"threads_started":1}
I20260929 17:56:18.448145 16801 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33709
I20260929 17:56:18.448375 16801 heartbeater.cc:464] Registering TS with master...
I20260929 17:56:18.448889 16801 heartbeater.cc:511] Master 127.0.98.62:33709 requested a full tablet report, sending...
I20260929 17:56:18.449963 16338 ts_manager.cc:194] Registered new tserver with Master: e8a734f480b34296ac1bb50572558f10 (127.0.98.3:34933)
I20260929 17:56:18.450611 16338 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:49559
I20260929 17:56:18.451040 16804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:56:18.443566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52256 (local address 127.0.98.62:33709)
0929 17:56:18.443773 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0929 17:56:18.443778 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:56:18.444147 (+ 369us) server_negotiation.cc:407] Connection header received
0929 17:56:18.444275 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:56:18.444278 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:56:18.444321 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:56:18.444390 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:56:18.445079 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:56:18.445595 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:56:18.446450 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:56:18.446727 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:56:18.446821 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:56:18.448200 (+ 1379us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:56:18.448203 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:56:18.448205 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:56:18.448229 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:56:18.449537 (+ 1308us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:56:18.449974 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:56:18.449977 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:56:18.449979 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:56:18.450060 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:56:18.450307 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:56:18.450311 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:56:18.450313 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:56:18.450497 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:56:18.450688 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0929 17:56:18.450784 (+ 96us) server_negotiation.cc:299] Negotiation successful
0929 17:56:18.450895 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":59,"threads_started":1}
I20260929 17:56:18.452572 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 17:56:18.455827 16804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:56:18.453109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52262 (local address 127.0.98.62:33709)
0929 17:56:18.453204 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0929 17:56:18.453208 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:56:18.453392 (+ 184us) server_negotiation.cc:407] Connection header received
0929 17:56:18.453499 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:56:18.453501 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:56:18.453581 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:56:18.453655 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 17:56:18.453800 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:56:18.454255 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:56:18.455005 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:56:18.455204 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:56:18.455267 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:56:18.455465 (+ 198us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 17:56:18.455577 (+ 112us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0929 17:56:18.455749 (+ 172us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:52262: token expired
Metrics: {"server-negotiator.queue_time_us":32}
W20260929 17:56:18.455878 16804 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:52262: token expired
W20260929 17:56:18.456168 16265 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.98.62:33709)
I20260929 17:56:18.461997 16804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:56:18.456455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52276 (local address 127.0.98.62:33709)
0929 17:56:18.456549 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0929 17:56:18.456553 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:56:18.456565 (+ 12us) server_negotiation.cc:407] Connection header received
0929 17:56:18.456636 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:56:18.456638 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:56:18.456667 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:56:18.456723 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:56:18.457252 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:56:18.457693 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:56:18.458395 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:56:18.458576 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:56:18.458637 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:56:18.459698 (+ 1061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:56:18.459701 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:56:18.459702 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:56:18.459717 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:56:18.460862 (+ 1145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:56:18.461224 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:56:18.461226 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:56:18.461227 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:56:18.461292 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:56:18.461470 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:56:18.461472 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:56:18.461473 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:56:18.461600 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:56:18.461687 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0929 17:56:18.461825 (+ 138us) server_negotiation.cc:299] Negotiation successful
0929 17:56:18.461901 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260929 17:56:18.464025 16338 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52276:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 17:56:18.467247 16338 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 17:56:18.474265 16599 tablet_service.cc:1467] Processing CreateTablet for tablet 91ac91019d494f8a9d8d9024b1de49b7 (DEFAULT_TABLE table=test-table [id=eb33037c41334e33a2ebf4fa4c5beb75]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 17:56:18.474265 16460 tablet_service.cc:1467] Processing CreateTablet for tablet 91ac91019d494f8a9d8d9024b1de49b7 (DEFAULT_TABLE table=test-table [id=eb33037c41334e33a2ebf4fa4c5beb75]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 17:56:18.474891 16460 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91ac91019d494f8a9d8d9024b1de49b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:56:18.474892 16599 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91ac91019d494f8a9d8d9024b1de49b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:56:18.477710 16557 tablet_bootstrap.cc:492] T 91ac91019d494f8a9d8d9024b1de49b7 P b499dd73acdb48028230a6e4f02bd2d4: Bootstrap starting.
I20260929 17:56:18.478113 16418 tablet_bootstrap.cc:492] T 91ac91019d494f8a9d8d9024b1de49b7 P 433a7f95a3694411b43c7a77d7fc93ea: Bootstrap starting.
I20260929 17:56:18.478428 16533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:56:18.473423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34933 (local address 127.0.0.1:60376)
0929 17:56:18.473515 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0929 17:56:18.473530 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 17:56:18.473605 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:56:18.474591 (+ 986us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:56:18.474594 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:56:18.474614 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:56:18.474725 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:56:18.474732 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:56:18.476042 (+ 1310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:56:18.476046 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:56:18.477079 (+ 1033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:56:18.477090 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:56:18.478032 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:56:18.478036 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:56:18.478128 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:56:18.478154 (+ 26us) client_negotiation.cc:769] Sending connection context
0929 17:56:18.478221 (+ 67us) client_negotiation.cc:240] Negotiation successful
0929 17:56:18.478303 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26}
I20260929 17:56:18.478606 16557 tablet_bootstrap.cc:654] T 91ac91019d494f8a9d8d9024b1de49b7 P b499dd73acdb48028230a6e4f02bd2d4: Neither blocks nor log segments found. Creating new log.
I20260929 17:56:18.478780 16810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:56:18.473699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60376 (local address 127.0.98.3:34933)
0929 17:56:18.473981 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0929 17:56:18.473994 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 17:56:18.474017 (+ 23us) server_negotiation.cc:407] Connection header received
0929 17:56:18.474098 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:56:18.474103 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:56:18.474359 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:56:18.474889 (+ 530us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:56:18.475098 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:56:18.475915 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:56:18.477195 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:56:18.477924 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:56:18.478032 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:56:18.478259 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0929 17:56:18.478314 (+ 55us) server_negotiation.cc:299] Negotiation successful
0929 17:56:18.478406 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":193,"thread_start_us":118,"threads_started":1}
I20260929 17:56:18.478914 16418 tablet_bootstrap.cc:654] T 91ac91019d494f8a9d8d9024b1de49b7 P 433a7f95a3694411b43c7a77d7fc93ea: Neither blocks nor log segments found. Creating new log.
I20260929 17:56:18.480167 16557 tablet_bootstrap.cc:492] T 91ac91019d494f8a9d8d9024b1de49b7 P b499dd73acdb48028230a6e4f02bd2d4: No bootstrap required, opened a new log
I20260929 17:56:18.480257 16557 ts_tablet_manager.cc:1397] T 91ac91019d494f8a9d8d9024b1de49b7 P b499dd73acdb48028230a6e4f02bd2d4: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260929 17:56:18.480633 16557 raft_consensus.cc:348] T 91ac91019d494f8a9d8d9024b1de49b7 P b499dd73acdb48028230a6e4f02bd2d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8a734f480b34296ac1bb50572558f10" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34933 } } peers { permanent_uuid: "433a7f95a3694411b43c7a77d7fc93ea" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46395 } } peers { permanent_uuid: "b499dd73acdb48028230a6e4f02bd2d4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33451 } }
I20260929 17:56:18.480728 16557 raft_consensus.cc:374] T 91ac91019d494f8a9d8d9024b1de49b7 P b499dd73acdb48028230a6e4f02bd2d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:56:18.480774 16557 raft_consensus.cc:729] T 91ac91019d494f8a9d8d9024b1de49b7 P b499dd73acdb48028230a6e4f02bd2d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b499dd73acdb48028230a6e4f02bd2d4, State: Initialized, Role: FOLLOWER
I20260929 17:56:18.480902 16557 consensus_queue.cc:260] T 91ac91019d494f8a9d8d9024b1de49b7 P b499dd73acdb48028230a6e4f02bd2d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8a734f480b34296ac1bb50572558f10" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34933 } } peers { permanent_uuid: "433a7f95a3694411b43c7a77d7fc93ea" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46395 } } peers { permanent_uuid: "b499dd73acdb48028230a6e4f02bd2d4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33451 } }
I20260929 17:56:18.481182 16557 ts_tablet_manager.cc:1428] T 91ac91019d494f8a9d8d9024b1de49b7 P b499dd73acdb48028230a6e4f02bd2d4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 17:56:18.481866 16418 tablet_bootstrap.cc:492] T 91ac91019d494f8a9d8d9024b1de49b7 P 433a7f95a3694411b43c7a77d7fc93ea: No bootstrap required, opened a new log
I20260929 17:56:18.481956 16418 ts_tablet_manager.cc:1397] T 91ac91019d494f8a9d8d9024b1de49b7 P 433a7f95a3694411b43c7a77d7fc93ea: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260929 17:56:18.482394 16418 raft_consensus.cc:348] T 91ac91019d494f8a9d8d9024b1de49b7 P 433a7f95a3694411b43c7a77d7fc93ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8a734f480b34296ac1bb50572558f10" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34933 } } peers { permanent_uuid: "433a7f95a3694411b43c7a77d7fc93ea" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46395 } } peers { permanent_uuid: "b499dd73acdb48028230a6e4f02bd2d4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33451 } }
I20260929 17:56:18.482515 16418 raft_consensus.cc:374] T 91ac91019d494f8a9d8d9024b1de49b7 P 433a7f95a3694411b43c7a77d7fc93ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:56:18.482556 16418 raft_consensus.cc:729] T 91ac91019d494f8a9d8d9024b1de49b7 P 433a7f95a3694411b43c7a77d7fc93ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 433a7f95a3694411b43c7a77d7fc93ea, State: Initialized, Role: FOLLOWER
I20260929 17:56:18.482676 16418 consensus_queue.cc:260] T 91ac91019d494f8a9d8d9024b1de49b7 P 433a7f95a3694411b43c7a77d7fc93ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8a734f480b34296ac1bb50572558f10" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34933 } } peers { permanent_uuid: "433a7f95a3694411b43c7a77d7fc93ea" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46395 } } peers { permanent_uuid: "b499dd73acdb48028230a6e4f02bd2d4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33451 } }
I20260929 17:56:18.482416 16735 tablet_service.cc:1467] Processing CreateTablet for tablet 91ac91019d494f8a9d8d9024b1de49b7 (DEFAULT_TABLE table=test-table [id=eb33037c41334e33a2ebf4fa4c5beb75]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 17:56:18.482918 16418 ts_tablet_manager.cc:1428] T 91ac91019d494f8a9d8d9024b1de49b7 P 433a7f95a3694411b43c7a77d7fc93ea: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260929 17:56:18.483608 16735 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91ac91019d494f8a9d8d9024b1de49b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:56:18.490568 16813 tablet_bootstrap.cc:492] T 91ac91019d494f8a9d8d9024b1de49b7 P e8a734f480b34296ac1bb50572558f10: Bootstrap starting.
I20260929 17:56:18.492554 16813 tablet_bootstrap.cc:654] T 91ac91019d494f8a9d8d9024b1de49b7 P e8a734f480b34296ac1bb50572558f10: Neither blocks nor log segments found. Creating new log.
I20260929 17:56:18.493604 16813 log.cc:824] T 91ac91019d494f8a9d8d9024b1de49b7 P e8a734f480b34296ac1bb50572558f10: Log is configured to *not* fsync() on all Append() calls
I20260929 17:56:18.495357 16813 tablet_bootstrap.cc:492] T 91ac91019d494f8a9d8d9024b1de49b7 P e8a734f480b34296ac1bb50572558f10: No bootstrap required, opened a new log
I20260929 17:56:18.495533 16813 ts_tablet_manager.cc:1397] T 91ac91019d494f8a9d8d9024b1de49b7 P e8a734f480b34296ac1bb50572558f10: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20260929 17:56:18.498564 16813 raft_consensus.cc:348] T 91ac91019d494f8a9d8d9024b1de49b7 P e8a734f480b34296ac1bb50572558f10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8a734f480b34296ac1bb50572558f10" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34933 } } peers { permanent_uuid: "433a7f95a3694411b43c7a77d7fc93ea" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46395 } } peers { permanent_uuid: "b499dd73acdb48028230a6e4f02bd2d4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33451 } }
I20260929 17:56:18.498804 16813 raft_consensus.cc:374] T 91ac91019d494f8a9d8d9024b1de49b7 P e8a734f480b34296ac1bb50572558f10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:56:18.498862 16813 raft_consensus.cc:729] T 91ac91019d494f8a9d8d9024b1de49b7 P e8a734f480b34296ac1bb50572558f10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8a734f480b34296ac1bb50572558f10, State: Initialized, Role: FOLLOWER
I20260929 17:56:18.499266 16813 consensus_queue.cc:260] T 91ac91019d494f8a9d8d9024b1de49b7 P e8a734f480b34296ac1bb50572558f10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8a734f480b34296ac1bb50572558f10" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34933 } } peers { permanent_uuid: "433a7f95a3694411b43c7a77d7fc93ea" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46395 } } peers { permanent_uuid: "b499dd73acdb48028230a6e4f02bd2d4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33451 } }
I20260929 17:56:18.499926 16801 heartbeater.cc:503] Master 127.0.98.62:33709 was elected leader, sending a full tablet report...
I20260929 17:56:18.500067 16532 raft_consensus.cc:482] T 91ac91019d494f8a9d8d9024b1de49b7 P 433a7f95a3694411b43c7a77d7fc93ea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 17:56:18.500196 16813 ts_tablet_manager.cc:1428] T 91ac91019d494f8a9d8d9024b1de49b7 P e8a734f480b34296ac1bb50572558f10: Time spent starting tablet: real 0.005s user 0.004s sys 0.001s
I20260929 17:56:18.500255 16532 raft_consensus.cc:504] T 91ac91019d494f8a9d8d9024b1de49b7 P 433a7f95a3694411b43c7a77d7fc93ea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8a734f480b34296ac1bb50572558f10" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34933 } } peers { permanent_uuid: "433a7f95a3694411b43c7a77d7fc93ea" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46395 } } peers { permanent_uuid: "b499dd73acdb48028230a6e4f02bd2d4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33451 } }
I20260929 17:56:18.501472 16532 leader_election.cc:288] T 91ac91019d494f8a9d8d9024b1de49b7 P 433a7f95a3694411b43c7a77d7fc93ea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e8a734f480b34296ac1bb50572558f10 (127.0.98.3:34933), b499dd73acdb48028230a6e4f02bd2d4 (127.0.98.2:33451)
W20260929 17:56:18.502168 16666 tablet.cc:2369] T 91ac91019d494f8a9d8d9024b1de49b7 P b499dd73acdb48028230a6e4f02bd2d4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 17:56:18.505851 16671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:56:18.502287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45867 (local address 127.0.98.2:33451)
0929 17:56:18.502398 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0929 17:56:18.502402 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:56:18.502416 (+ 14us) server_negotiation.cc:407] Connection header received
0929 17:56:18.502465 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:56:18.502468 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:56:18.502514 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:56:18.502584 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:56:18.502892 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:56:18.503574 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:56:18.504715 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:56:18.505292 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:56:18.505380 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:56:18.505430 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0929 17:56:18.505657 (+ 227us) server_negotiation.cc:299] Negotiation successful
0929 17:56:18.505730 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 17:56:18.505852 16816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:56:18.501905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33451 (local address 127.0.98.1:45867)
0929 17:56:18.502226 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0929 17:56:18.502238 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 17:56:18.502329 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:56:18.502630 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:56:18.502632 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:56:18.502663 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:56:18.502794 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:56:18.502802 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:56:18.503680 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:56:18.503683 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:56:18.504615 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:56:18.504626 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:56:18.505417 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:56:18.505422 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:56:18.505512 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:56:18.505564 (+ 52us) client_negotiation.cc:769] Sending connection context
0929 17:56:18.505634 (+ 70us) client_negotiation.cc:240] Negotiation successful
0929 17:56:18.505707 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"mutex_wait_us":38,"thread_start_us":120,"threads_started":1}
I20260929 17:56:18.505985 16810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:56:18.501751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46377 (local address 127.0.98.3:34933)
0929 17:56:18.501985 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0929 17:56:18.501990 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:56:18.502604 (+ 614us) server_negotiation.cc:407] Connection header received
0929 17:56:18.502644 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:56:18.502647 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:56:18.502688 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:56:18.502753 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:56:18.503044 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:56:18.503862 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:56:18.504965 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:56:18.505498 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:56:18.505595 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:56:18.505659 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0929 17:56:18.505818 (+ 159us) server_negotiation.cc:299] Negotiation successful
0929 17:56:18.505887 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260929 17:56:18.505991 16528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:56:18.501644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34933 (local address 127.0.98.1:46377)
0929 17:56:18.501778 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0929 17:56:18.501794 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 17:56:18.501894 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:56:18.502778 (+ 884us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:56:18.502781 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:56:18.502798 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:56:18.502929 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:56:18.502936 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:56:18.503968 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:56:18.503972 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:56:18.504843 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:56:18.504853 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:56:18.505605 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:56:18.505609 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:56:18.505704 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:56:18.505724 (+ 20us) client_negotiation.cc:769] Sending connection context
0929 17:56:18.505798 (+ 74us) client_negotiation.cc:240] Negotiation successful
0929 17:56:18.505874 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57,"mutex_wait_us":39}
I20260929 17:56:18.506541 16755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91ac91019d494f8a9d8d9024b1de49b7" candidate_uuid: "433a7f95a3694411b43c7a77d7fc93ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8a734f480b34296ac1bb50572558f10" is_pre_election: true
I20260929 17:56:18.506702 16619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91ac91019d494f8a9d8d9024b1de49b7" candidate_uuid: "433a7f95a3694411b43c7a77d7fc93ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b499dd73acdb48028230a6e4f02bd2d4" is_pre_election: true
I20260929 17:56:18.506809 16755 raft_consensus.cc:2454] T 91ac91019d494f8a9d8d9024b1de49b7 P e8a734f480b34296ac1bb50572558f10 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 433a7f95a3694411b43c7a77d7fc93ea in term 0.
I20260929 17:56:18.506985 16619 raft_consensus.cc:2454] T 91ac91019d494f8a9d8d9024b1de49b7 P b499dd73acdb48028230a6e4f02bd2d4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 433a7f95a3694411b43c7a77d7fc93ea in term 0.
I20260929 17:56:18.507285 16412 leader_election.cc:302] T 91ac91019d494f8a9d8d9024b1de49b7 P 433a7f95a3694411b43c7a77d7fc93ea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 433a7f95a3694411b43c7a77d7fc93ea, e8a734f480b34296ac1bb50572558f10; no voters:
I20260929 17:56:18.507516 16532 raft_consensus.cc:2781] T 91ac91019d494f8a9d8d9024b1de49b7 P 433a7f95a3694411b43c7a77d7fc93ea [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 17:56:18.507591 16532 raft_consensus.cc:482] T 91ac91019d494f8a9d8d9024b1de49b7 P 433a7f95a3694411b43c7a77d7fc93ea [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 17:56:18.507640 16532 raft_consensus.cc:3037] T 91ac91019d494f8a9d8d9024b1de49b7 P 433a7f95a3694411b43c7a77d7fc93ea [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:56:18.508812 16532 raft_consensus.cc:504] T 91ac91019d494f8a9d8d9024b1de49b7 P 433a7f95a3694411b43c7a77d7fc93ea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8a734f480b34296ac1bb50572558f10" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34933 } } peers { permanent_uuid: "433a7f95a3694411b43c7a77d7fc93ea" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46395 } } peers { permanent_uuid: "b499dd73acdb48028230a6e4f02bd2d4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33451 } }
I20260929 17:56:18.509166 16532 leader_election.cc:288] T 91ac91019d494f8a9d8d9024b1de49b7 P 433a7f95a3694411b43c7a77d7fc93ea [CANDIDATE]: Term 1 election: Requested vote from peers e8a734f480b34296ac1bb50572558f10 (127.0.98.3:34933), b499dd73acdb48028230a6e4f02bd2d4 (127.0.98.2:33451)
I20260929 17:56:18.509519 16755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91ac91019d494f8a9d8d9024b1de49b7" candidate_uuid: "433a7f95a3694411b43c7a77d7fc93ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8a734f480b34296ac1bb50572558f10"
I20260929 17:56:18.509591 16619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91ac91019d494f8a9d8d9024b1de49b7" candidate_uuid: "433a7f95a3694411b43c7a77d7fc93ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b499dd73acdb48028230a6e4f02bd2d4"
I20260929 17:56:18.509622 16755 raft_consensus.cc:3037] T 91ac91019d494f8a9d8d9024b1de49b7 P e8a734f480b34296ac1bb50572558f10 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:56:18.509720 16619 raft_consensus.cc:3037] T 91ac91019d494f8a9d8d9024b1de49b7 P b499dd73acdb48028230a6e4f02bd2d4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:56:18.510596 16755 raft_consensus.cc:2454] T 91ac91019d494f8a9d8d9024b1de49b7 P e8a734f480b34296ac1bb50572558f10 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 433a7f95a3694411b43c7a77d7fc93ea in term 1.
I20260929 17:56:18.510746 16619 raft_consensus.cc:2454] T 91ac91019d494f8a9d8d9024b1de49b7 P b499dd73acdb48028230a6e4f02bd2d4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 433a7f95a3694411b43c7a77d7fc93ea in term 1.
I20260929 17:56:18.510871 16412 leader_election.cc:302] T 91ac91019d494f8a9d8d9024b1de49b7 P 433a7f95a3694411b43c7a77d7fc93ea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 433a7f95a3694411b43c7a77d7fc93ea, e8a734f480b34296ac1bb50572558f10; no voters:
I20260929 17:56:18.511040 16532 raft_consensus.cc:2781] T 91ac91019d494f8a9d8d9024b1de49b7 P 433a7f95a3694411b43c7a77d7fc93ea [term 1 FOLLOWER]: Leader election won for term 1
I20260929 17:56:18.511230 16532 raft_consensus.cc:686] T 91ac91019d494f8a9d8d9024b1de49b7 P 433a7f95a3694411b43c7a77d7fc93ea [term 1 LEADER]: Becoming Leader. State: Replica: 433a7f95a3694411b43c7a77d7fc93ea, State: Running, Role: LEADER
I20260929 17:56:18.511471 16532 consensus_queue.cc:237] T 91ac91019d494f8a9d8d9024b1de49b7 P 433a7f95a3694411b43c7a77d7fc93ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8a734f480b34296ac1bb50572558f10" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34933 } } peers { permanent_uuid: "433a7f95a3694411b43c7a77d7fc93ea" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46395 } } peers { permanent_uuid: "b499dd73acdb48028230a6e4f02bd2d4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33451 } }
I20260929 17:56:18.514312 16338 catalog_manager.cc:5495] T 91ac91019d494f8a9d8d9024b1de49b7 P 433a7f95a3694411b43c7a77d7fc93ea reported cstate change: term changed from 0 to 1, leader changed from <none> to 433a7f95a3694411b43c7a77d7fc93ea (127.0.98.1). New cstate: current_term: 1 leader_uuid: "433a7f95a3694411b43c7a77d7fc93ea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8a734f480b34296ac1bb50572558f10" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34933 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "433a7f95a3694411b43c7a77d7fc93ea" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46395 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b499dd73acdb48028230a6e4f02bd2d4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33451 } health_report { overall_health: UNKNOWN } } }
I20260929 17:56:18.531337 16534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:56:18.528051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47370 (local address 127.0.98.1:46395)
0929 17:56:18.528171 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0929 17:56:18.528174 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 17:56:18.528188 (+ 14us) server_negotiation.cc:407] Connection header received
0929 17:56:18.528363 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:56:18.528365 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:56:18.528417 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:56:18.528503 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 17:56:18.528758 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:56:18.529427 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:56:18.530189 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:56:18.530413 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:56:18.530479 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:56:18.530658 (+ 179us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 17:56:18.531061 (+ 403us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 17:56:18.531143 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0929 17:56:18.531188 (+ 45us) server_negotiation.cc:299] Negotiation successful
0929 17:56:18.531233 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260929 17:56:18.535059 16619 raft_consensus.cc:1261] T 91ac91019d494f8a9d8d9024b1de49b7 P b499dd73acdb48028230a6e4f02bd2d4 [term 1 FOLLOWER]: Refusing update from remote peer 433a7f95a3694411b43c7a77d7fc93ea: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 17:56:18.535063 16755 raft_consensus.cc:1261] T 91ac91019d494f8a9d8d9024b1de49b7 P e8a734f480b34296ac1bb50572558f10 [term 1 FOLLOWER]: Refusing update from remote peer 433a7f95a3694411b43c7a77d7fc93ea: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 17:56:18.535696 16817 consensus_queue.cc:1035] T 91ac91019d494f8a9d8d9024b1de49b7 P 433a7f95a3694411b43c7a77d7fc93ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8a734f480b34296ac1bb50572558f10" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 17:56:18.535879 16532 consensus_queue.cc:1035] T 91ac91019d494f8a9d8d9024b1de49b7 P 433a7f95a3694411b43c7a77d7fc93ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "b499dd73acdb48028230a6e4f02bd2d4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 17:56:18.542073 16820 mvcc.cc:205] Tried to move back new op lower bound from 7334725953673494528 to 7334725953585156096. Current Snapshot: MvccSnapshot[applied={T|T < 7334725953673494528}]
I20260929 17:56:18.543754 16821 mvcc.cc:205] Tried to move back new op lower bound from 7334725953673494528 to 7334725953585156096. Current Snapshot: MvccSnapshot[applied={T|T < 7334725953673494528}]
I20260929 17:56:18.549889 16338 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52276:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 17:56:18.550088 16338 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52276:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 17:56:18.552544 16338 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 842b44590ee242fca73034b1300da182: Sending DeleteTablet for 3 replicas of tablet 91ac91019d494f8a9d8d9024b1de49b7
I20260929 17:56:18.553222 16460 tablet_service.cc:1514] Processing DeleteTablet for tablet 91ac91019d494f8a9d8d9024b1de49b7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 17:56:18 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55444
I20260929 17:56:18.553299 16735 tablet_service.cc:1514] Processing DeleteTablet for tablet 91ac91019d494f8a9d8d9024b1de49b7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 17:56:18 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60376
I20260929 17:56:18.553421 16535 tablet_replica.cc:330] T 91ac91019d494f8a9d8d9024b1de49b7 P 433a7f95a3694411b43c7a77d7fc93ea: stopping tablet replica
I20260929 17:56:18.553560 16535 raft_consensus.cc:2229] T 91ac91019d494f8a9d8d9024b1de49b7 P 433a7f95a3694411b43c7a77d7fc93ea [term 1 LEADER]: Raft consensus shutting down.
I20260929 17:56:18.553854 16535 raft_consensus.cc:2258] T 91ac91019d494f8a9d8d9024b1de49b7 P 433a7f95a3694411b43c7a77d7fc93ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 17:56:18.553879 16599 tablet_service.cc:1514] Processing DeleteTablet for tablet 91ac91019d494f8a9d8d9024b1de49b7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 17:56:18 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37164
I20260929 17:56:18.553977 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskup3Soe/build/debug/bin/kudu with pid 16376
I20260929 17:56:18.554481 16672 tablet_replica.cc:330] T 91ac91019d494f8a9d8d9024b1de49b7 P b499dd73acdb48028230a6e4f02bd2d4: stopping tablet replica
I20260929 17:56:18.554591 16672 raft_consensus.cc:2229] T 91ac91019d494f8a9d8d9024b1de49b7 P b499dd73acdb48028230a6e4f02bd2d4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 17:56:18.554726 16672 raft_consensus.cc:2258] T 91ac91019d494f8a9d8d9024b1de49b7 P b499dd73acdb48028230a6e4f02bd2d4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 17:56:18.555485 16830 tablet_replica.cc:330] T 91ac91019d494f8a9d8d9024b1de49b7 P e8a734f480b34296ac1bb50572558f10: stopping tablet replica
I20260929 17:56:18.555639 16672 ts_tablet_manager.cc:1905] T 91ac91019d494f8a9d8d9024b1de49b7 P b499dd73acdb48028230a6e4f02bd2d4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 17:56:18.555831 16830 raft_consensus.cc:2229] T 91ac91019d494f8a9d8d9024b1de49b7 P e8a734f480b34296ac1bb50572558f10 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 17:56:18.556159 16830 raft_consensus.cc:2258] T 91ac91019d494f8a9d8d9024b1de49b7 P e8a734f480b34296ac1bb50572558f10 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 17:56:18.557876 16672 ts_tablet_manager.cc:1918] T 91ac91019d494f8a9d8d9024b1de49b7 P b499dd73acdb48028230a6e4f02bd2d4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 17:56:18.557947 16672 log.cc:1196] T 91ac91019d494f8a9d8d9024b1de49b7 P b499dd73acdb48028230a6e4f02bd2d4: Deleting WAL directory at /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/minicluster-data/ts-1/wal/wals/91ac91019d494f8a9d8d9024b1de49b7
I20260929 17:56:18.558209 16672 ts_tablet_manager.cc:1939] T 91ac91019d494f8a9d8d9024b1de49b7 P b499dd73acdb48028230a6e4f02bd2d4: Deleting consensus metadata
I20260929 17:56:18.558885 16322 catalog_manager.cc:4841] TS b499dd73acdb48028230a6e4f02bd2d4 (127.0.98.2:33451): tablet 91ac91019d494f8a9d8d9024b1de49b7 (table test-table [id=eb33037c41334e33a2ebf4fa4c5beb75]) successfully deleted
I20260929 17:56:18.559365 16830 ts_tablet_manager.cc:1905] T 91ac91019d494f8a9d8d9024b1de49b7 P e8a734f480b34296ac1bb50572558f10: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260929 17:56:18.561053 16323 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:46395 (error 108)
I20260929 17:56:18.561269 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskup3Soe/build/debug/bin/kudu with pid 16530
I20260929 17:56:18.562330 16533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:56:18.562066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46395 (local address 127.0.0.1:47380)
0929 17:56:18.562199 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0929 17:56:18.562262 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260929 17:56:18.562526 16323 catalog_manager.cc:4584] TS 433a7f95a3694411b43c7a77d7fc93ea (127.0.98.1:46395): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 91ac91019d494f8a9d8d9024b1de49b7: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46395: connect: Connection refused (error 111)
I20260929 17:56:18.565857 16830 ts_tablet_manager.cc:1918] T 91ac91019d494f8a9d8d9024b1de49b7 P e8a734f480b34296ac1bb50572558f10: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 17:56:18.565951 16830 log.cc:1196] T 91ac91019d494f8a9d8d9024b1de49b7 P e8a734f480b34296ac1bb50572558f10: Deleting WAL directory at /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/minicluster-data/ts-2/wal/wals/91ac91019d494f8a9d8d9024b1de49b7
I20260929 17:56:18.566226 16830 ts_tablet_manager.cc:1939] T 91ac91019d494f8a9d8d9024b1de49b7 P e8a734f480b34296ac1bb50572558f10: Deleting consensus metadata
I20260929 17:56:18.567225 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskup3Soe/build/debug/bin/kudu with pid 16668
I20260929 17:56:18.567970 16322 catalog_manager.cc:4841] TS e8a734f480b34296ac1bb50572558f10 (127.0.98.3:34933): tablet 91ac91019d494f8a9d8d9024b1de49b7 (table test-table [id=eb33037c41334e33a2ebf4fa4c5beb75]) successfully deleted
I20260929 17:56:18.572428 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskup3Soe/build/debug/bin/kudu with pid 16305
I20260929 17:56:38.578059 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskup3Soe/build/debug/bin/kudu
/tmp/dist-test-taskup3Soe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33709
--webserver_interface=127.0.98.62
--webserver_port=43621
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33709
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:56:38.689283 16832 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:56:38.689604 16832 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:56:38.689666 16832 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:56:38.693871 16832 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260929 17:56:38.693928 16832 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260929 17:56:38.693949 16832 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260929 17:56:38.693967 16832 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 17:56:38.693990 16832 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:56:38.694015 16832 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:56:38.694046 16832 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 17:56:38.694075 16832 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 17:56:38.699676 16832 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33709
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33709
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=43621
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16832
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:56:38.700802 16832 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:56:38.701834 16832 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 17:56:38.710420 16832 server_base.cc:1034] running on GCE node
W20260929 17:56:38.710371 16837 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:56:38.710379 16838 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:56:38.710342 16840 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:56:38.711236 16832 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:56:38.712417 16832 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:56:38.713621 16832 hybrid_clock.cc:648] HybridClock initialized: now 1790704598713600 us; error 36 us; skew 500 ppm
Sep 29 17:56:38 dist-test-slave-53nh krb5kdc[15758](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704598, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:56:38.722371 16832 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 17:56:38.723421 16832 webserver.cc:460] Webserver started at http://127.0.98.62:43621/ using document root <none> and password file <none>
I20260929 17:56:38.723996 16832 fs_manager.cc:359] Metadata directory not provided
I20260929 17:56:38.724068 16832 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:56:38.728124 16832 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260929 17:56:38.730692 16847 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:56:38.732101 16832 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 17:56:38.732266 16832 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/minicluster-data/master-0/wal
uuid: "842b44590ee242fca73034b1300da182"
format_stamp: "Formatted at 2026-09-29 17:55:56 on dist-test-slave-53nh"
server_key: "8021b0090bb6c1c80b9b0f788960a2e6"
server_key_iv: "f505f861ff6bb656c97c8657b4915be4"
server_key_version: "encryptionkey@0"
I20260929 17:56:38.732888 16832 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:56:38.752195 16832 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:56:38.755831 16832 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:56:38.756057 16832 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:56:38.763361 16832 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33709
I20260929 17:56:38.763716 16899 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33709 every 8 connection(s)
I20260929 17:56:38.764681 16832 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/minicluster-data/master-0/data/info.pb
I20260929 17:56:38.769212 16900 sys_catalog.cc:263] Verifying existing consensus state
I20260929 17:56:38.770919 16900 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 842b44590ee242fca73034b1300da182: Bootstrap starting.
I20260929 17:56:38.774915 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskup3Soe/build/debug/bin/kudu as pid 16832
I20260929 17:56:38.775322 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskup3Soe/build/debug/bin/kudu
/tmp/dist-test-taskup3Soe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:46395
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:33709
--webserver_port=43617
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 17:56:38.788805 16900 log.cc:824] T 00000000000000000000000000000000 P 842b44590ee242fca73034b1300da182: Log is configured to *not* fsync() on all Append() calls
I20260929 17:56:38.800608 16900 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 842b44590ee242fca73034b1300da182: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 17:56:38.800987 16900 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 842b44590ee242fca73034b1300da182: Bootstrap complete.
I20260929 17:56:38.804903 16900 raft_consensus.cc:348] T 00000000000000000000000000000000 P 842b44590ee242fca73034b1300da182 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "842b44590ee242fca73034b1300da182" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33709 } }
I20260929 17:56:38.805378 16900 raft_consensus.cc:729] T 00000000000000000000000000000000 P 842b44590ee242fca73034b1300da182 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 842b44590ee242fca73034b1300da182, State: Initialized, Role: FOLLOWER
I20260929 17:56:38.805872 16900 consensus_queue.cc:260] T 00000000000000000000000000000000 P 842b44590ee242fca73034b1300da182 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "842b44590ee242fca73034b1300da182" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33709 } }
I20260929 17:56:38.806012 16900 raft_consensus.cc:388] T 00000000000000000000000000000000 P 842b44590ee242fca73034b1300da182 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 17:56:38.806082 16900 raft_consensus.cc:482] T 00000000000000000000000000000000 P 842b44590ee242fca73034b1300da182 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 17:56:38.806177 16900 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 842b44590ee242fca73034b1300da182 [term 2 FOLLOWER]: Advancing to term 3
I20260929 17:56:38.808060 16900 raft_consensus.cc:504] T 00000000000000000000000000000000 P 842b44590ee242fca73034b1300da182 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "842b44590ee242fca73034b1300da182" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33709 } }
I20260929 17:56:38.808377 16900 leader_election.cc:302] T 00000000000000000000000000000000 P 842b44590ee242fca73034b1300da182 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 842b44590ee242fca73034b1300da182; no voters:
I20260929 17:56:38.808667 16900 leader_election.cc:288] T 00000000000000000000000000000000 P 842b44590ee242fca73034b1300da182 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260929 17:56:38.808765 16904 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 842b44590ee242fca73034b1300da182 [term 3 FOLLOWER]: Leader election won for term 3
I20260929 17:56:38.808950 16904 raft_consensus.cc:686] T 00000000000000000000000000000000 P 842b44590ee242fca73034b1300da182 [term 3 LEADER]: Becoming Leader. State: Replica: 842b44590ee242fca73034b1300da182, State: Running, Role: LEADER
I20260929 17:56:38.809297 16904 consensus_queue.cc:237] T 00000000000000000000000000000000 P 842b44590ee242fca73034b1300da182 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "842b44590ee242fca73034b1300da182" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33709 } }
I20260929 17:56:38.809933 16900 sys_catalog.cc:564] T 00000000000000000000000000000000 P 842b44590ee242fca73034b1300da182 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 17:56:38.810273 16906 sys_catalog.cc:455] T 00000000000000000000000000000000 P 842b44590ee242fca73034b1300da182 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 842b44590ee242fca73034b1300da182. Latest consensus state: current_term: 3 leader_uuid: "842b44590ee242fca73034b1300da182" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "842b44590ee242fca73034b1300da182" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33709 } } }
I20260929 17:56:38.810400 16906 sys_catalog.cc:458] T 00000000000000000000000000000000 P 842b44590ee242fca73034b1300da182 [sys.catalog]: This master's current role is: LEADER
I20260929 17:56:38.810706 16905 sys_catalog.cc:455] T 00000000000000000000000000000000 P 842b44590ee242fca73034b1300da182 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "842b44590ee242fca73034b1300da182" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "842b44590ee242fca73034b1300da182" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33709 } } }
I20260929 17:56:38.810803 16905 sys_catalog.cc:458] T 00000000000000000000000000000000 P 842b44590ee242fca73034b1300da182 [sys.catalog]: This master's current role is: LEADER
I20260929 17:56:38.811810 16908 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 17:56:38.816956 16908 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 17:56:38.817610 16908 catalog_manager.cc:1259] Loaded cluster ID: 92f56c901b26449fbe079d5de709dbdd
I20260929 17:56:38.817670 16908 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 17:56:38.820737 16908 catalog_manager.cc:1504] Loading token signing keys...
I20260929 17:56:38.821587 16908 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 842b44590ee242fca73034b1300da182: Loaded TSK: 0
I20260929 17:56:38.821671 16908 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 842b44590ee242fca73034b1300da182: Loaded TSK: 1
F20260929 17:56:38.822360 16908 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7efe31c689ad google::LogMessage::Fail() at ??:0
@ 0x7efe31c6b88a google::LogMessage::SendToLog() at ??:0
@ 0x7efe31c684de google::LogMessage::Flush() at ??:0
@ 0x7efe31c69a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7efe3492c18d kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7efe34924f0d kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7efe3c5cd7bb kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7efe3c5a799c kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7efe3c5a8178 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlvE3_clEv at ??:0
@ 0x7efe3c5e9df8 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7efe3c60882b std::function<>::operator()() at ??:0
@ 0x7efe3c5a7b4f _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7efe3c5a8f3e kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7efe3c5a5579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7efe3c5e9640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7efe33eba3d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7efe33ebac83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7efe33ebc3f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7efe33eab512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7efe354ef6db start_thread at ??:0
@ 0x7efe3076661f clone at ??:0
W20260929 17:56:38.922195 16902 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:56:38.922484 16902 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:56:38.922529 16902 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:56:38.926842 16902 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:56:38.926913 16902 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:56:38.926987 16902 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 17:56:38.932514 16902 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:46395
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=43617
--tserver_master_addrs=127.0.98.62:33709
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16902
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:56:38.933735 16902 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:56:38.934772 16902 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:56:38.944504 16947 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:56:38.944509 16950 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:56:38.944504 16948 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:56:38.944945 16902 server_base.cc:1034] running on GCE node
I20260929 17:56:38.945412 16902 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:56:38.946100 16902 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:56:38.947288 16902 hybrid_clock.cc:648] HybridClock initialized: now 1790704598947257 us; error 44 us; skew 500 ppm
Sep 29 17:56:38 dist-test-slave-53nh krb5kdc[15758](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704598, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:56:38.950368 16902 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260929 17:56:38.951344 16902 webserver.cc:460] Webserver started at http://127.0.98.1:43617/ using document root <none> and password file <none>
I20260929 17:56:38.951905 16902 fs_manager.cc:359] Metadata directory not provided
I20260929 17:56:38.951967 16902 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:56:38.956193 16902 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.002s
I20260929 17:56:38.958642 16957 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:56:38.959995 16902 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 17:56:38.960115 16902 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/minicluster-data/ts-0/wal
uuid: "433a7f95a3694411b43c7a77d7fc93ea"
format_stamp: "Formatted at 2026-09-29 17:55:57 on dist-test-slave-53nh"
server_key: "232988f4455d1b6900d092f246158ce9"
server_key_iv: "c2e13854c1871df45f647fde8765fdfa"
server_key_version: "encryptionkey@0"
I20260929 17:56:38.960708 16902 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:56:38.989779 16902 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:56:38.993338 16902 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:56:38.993517 16902 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:56:38.994164 16902 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 17:56:38.995362 16964 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260929 17:56:39.001333 16902 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260929 17:56:39.001423 16902 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.001s sys 0.000s
I20260929 17:56:39.001497 16902 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260929 17:56:39.003260 16902 ts_tablet_manager.cc:610] Registered 1 tablets
I20260929 17:56:39.003319 16902 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260929 17:56:39.003691 16964 tablet_bootstrap.cc:492] T 91ac91019d494f8a9d8d9024b1de49b7 P 433a7f95a3694411b43c7a77d7fc93ea: Bootstrap starting.
I20260929 17:56:39.013206 16902 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46395
I20260929 17:56:39.013630 17071 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46395 every 8 connection(s)
I20260929 17:56:39.014412 16902 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/minicluster-data/ts-0/data/info.pb
I20260929 17:56:39.022619 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskup3Soe/build/debug/bin/kudu as pid 16902
I20260929 17:56:39.023397 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskup3Soe/build/debug/bin/kudu
/tmp/dist-test-taskup3Soe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:33451
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:33709
--webserver_port=35103
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 17:56:39.026170 16964 log.cc:824] T 91ac91019d494f8a9d8d9024b1de49b7 P 433a7f95a3694411b43c7a77d7fc93ea: Log is configured to *not* fsync() on all Append() calls
I20260929 17:56:39.036516 16964 tablet_bootstrap.cc:492] T 91ac91019d494f8a9d8d9024b1de49b7 P 433a7f95a3694411b43c7a77d7fc93ea: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 17:56:39.037029 16964 tablet_bootstrap.cc:492] T 91ac91019d494f8a9d8d9024b1de49b7 P 433a7f95a3694411b43c7a77d7fc93ea: Bootstrap complete.
I20260929 17:56:39.038311 16964 ts_tablet_manager.cc:1397] T 91ac91019d494f8a9d8d9024b1de49b7 P 433a7f95a3694411b43c7a77d7fc93ea: Time spent bootstrapping tablet: real 0.035s user 0.028s sys 0.005s
I20260929 17:56:39.041918 16964 raft_consensus.cc:348] T 91ac91019d494f8a9d8d9024b1de49b7 P 433a7f95a3694411b43c7a77d7fc93ea [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8a734f480b34296ac1bb50572558f10" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34933 } } peers { permanent_uuid: "433a7f95a3694411b43c7a77d7fc93ea" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46395 } } peers { permanent_uuid: "b499dd73acdb48028230a6e4f02bd2d4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33451 } }
I20260929 17:56:39.042542 16964 raft_consensus.cc:729] T 91ac91019d494f8a9d8d9024b1de49b7 P 433a7f95a3694411b43c7a77d7fc93ea [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 433a7f95a3694411b43c7a77d7fc93ea, State: Initialized, Role: FOLLOWER
I20260929 17:56:39.043148 16964 consensus_queue.cc:260] T 91ac91019d494f8a9d8d9024b1de49b7 P 433a7f95a3694411b43c7a77d7fc93ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8a734f480b34296ac1bb50572558f10" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34933 } } peers { permanent_uuid: "433a7f95a3694411b43c7a77d7fc93ea" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46395 } } peers { permanent_uuid: "b499dd73acdb48028230a6e4f02bd2d4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33451 } }
I20260929 17:56:39.044484 16964 ts_tablet_manager.cc:1428] T 91ac91019d494f8a9d8d9024b1de49b7 P 433a7f95a3694411b43c7a77d7fc93ea: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
W20260929 17:56:39.058073 17074 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:56:39.016390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33709 (local address 127.0.98.1:49243)
0929 17:56:39.016872 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0929 17:56:39.016926 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0929 17:56:39.017793 (+ 867us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:56:39.057279 (+ 39486us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33709: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":123,"threads_started":1}
W20260929 17:56:39.058585 17072 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:33709 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:33709: Client connection negotiation failed: client connection to 127.0.98.62:33709: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260929 17:56:39.059326 17074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:56:39.059070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33709 (local address 127.0.98.1:50669)
0929 17:56:39.059195 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0929 17:56:39.059260 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260929 17:56:39.059935 17074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:56:39.059728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33709 (local address 127.0.98.1:47789)
0929 17:56:39.059835 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0929 17:56:39.059886 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260929 17:56:39.060091 17072 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 17:56:39.140614 17075 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:56:39.140887 17075 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:56:39.140950 17075 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:56:39.145114 17075 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:56:39.145192 17075 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:56:39.145278 17075 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 17:56:39.150767 17075 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:33451
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=35103
--tserver_master_addrs=127.0.98.62:33709
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17075
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:56:39.151971 17075 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:56:39.153389 17075 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 17:56:39.163787 17075 server_base.cc:1034] running on GCE node
W20260929 17:56:39.163743 17082 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:56:39.163946 17083 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:56:39.163966 17085 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:56:39.164487 17075 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:56:39.165362 17075 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:56:39.166568 17075 hybrid_clock.cc:648] HybridClock initialized: now 1790704599166532 us; error 60 us; skew 500 ppm
Sep 29 17:56:39 dist-test-slave-53nh krb5kdc[15758](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704599, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:56:39.169492 17075 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260929 17:56:39.170356 17075 webserver.cc:460] Webserver started at http://127.0.98.2:35103/ using document root <none> and password file <none>
I20260929 17:56:39.170923 17075 fs_manager.cc:359] Metadata directory not provided
I20260929 17:56:39.170987 17075 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:56:39.174965 17075 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 17:56:39.177237 17092 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:56:39.178504 17075 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 17:56:39.178613 17075 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/minicluster-data/ts-1/wal
uuid: "b499dd73acdb48028230a6e4f02bd2d4"
format_stamp: "Formatted at 2026-09-29 17:55:57 on dist-test-slave-53nh"
server_key: "d2473c82904daeddcebaac9867c45400"
server_key_iv: "1943549757123a78e76af5f53abfd36d"
server_key_version: "encryptionkey@0"
I20260929 17:56:39.179163 17075 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:56:39.191376 17075 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:56:39.194816 17075 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:56:39.195013 17075 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:56:39.195664 17075 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 17:56:39.196585 17075 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:56:39.196632 17075 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:56:39.196673 17075 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:56:39.196694 17075 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 17:56:39.205235 17075 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33451
I20260929 17:56:39.205266 17205 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33451 every 8 connection(s)
I20260929 17:56:39.206312 17075 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/minicluster-data/ts-1/data/info.pb
I20260929 17:56:39.210346 17208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:56:39.208475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33709 (local address 127.0.98.2:42231)
0929 17:56:39.208958 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0929 17:56:39.209143 (+ 185us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":113,"threads_started":1}
W20260929 17:56:39.210808 17206 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:33709 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:33709: Client connection negotiation failed: client connection to 127.0.98.62:33709: connect: Connection refused (error 111)
I20260929 17:56:39.211423 17208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:56:39.211209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33709 (local address 127.0.98.2:40801)
0929 17:56:39.211318 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0929 17:56:39.211368 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260929 17:56:39.212059 17208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:56:39.211836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33709 (local address 127.0.98.2:35571)
0929 17:56:39.211966 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0929 17:56:39.212010 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260929 17:56:39.212251 17206 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260929 17:56:39.212504 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskup3Soe/build/debug/bin/kudu as pid 17075
I20260929 17:56:39.212899 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskup3Soe/build/debug/bin/kudu
/tmp/dist-test-taskup3Soe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:34933
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:33709
--webserver_port=46089
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:56:39.327452 17209 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:56:39.327790 17209 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:56:39.327836 17209 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:56:39.332073 17209 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:56:39.332154 17209 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:56:39.332227 17209 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 17:56:39.337914 17209 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:34933
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=46089
--tserver_master_addrs=127.0.98.62:33709
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17209
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:56:39.339053 17209 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:56:39.340103 17209 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 17:56:39.350593 17209 server_base.cc:1034] running on GCE node
W20260929 17:56:39.350484 17215 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:56:39.350545 17214 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:56:39.350435 17217 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:56:39.351280 17209 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:56:39.352015 17209 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:56:39.353223 17209 hybrid_clock.cc:648] HybridClock initialized: now 1790704599353208 us; error 32 us; skew 500 ppm
Sep 29 17:56:39 dist-test-slave-53nh krb5kdc[15758](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704599, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:56:39.356333 17209 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260929 17:56:39.357542 17209 webserver.cc:460] Webserver started at http://127.0.98.3:46089/ using document root <none> and password file <none>
I20260929 17:56:39.358179 17209 fs_manager.cc:359] Metadata directory not provided
I20260929 17:56:39.358259 17209 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:56:39.362763 17209 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 17:56:39.365250 17224 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:56:39.366672 17209 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 17:56:39.366799 17209 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/minicluster-data/ts-2/wal
uuid: "e8a734f480b34296ac1bb50572558f10"
format_stamp: "Formatted at 2026-09-29 17:55:57 on dist-test-slave-53nh"
server_key: "43d0aeb3bdff69a5cbc3dca0926f9313"
server_key_iv: "d6083adc1f9f419fab5de7ef217989b3"
server_key_version: "encryptionkey@0"
I20260929 17:56:39.367389 17209 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:56:39.378757 17209 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:56:39.381987 17209 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:56:39.382174 17209 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:56:39.382750 17209 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 17:56:39.383589 17209 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:56:39.383635 17209 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:56:39.383687 17209 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:56:39.383713 17209 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 17:56:39.392169 17209 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34933
I20260929 17:56:39.392225 17337 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34933 every 8 connection(s)
I20260929 17:56:39.393245 17209 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790704375975685-392-0/minicluster-data/ts-2/data/info.pb
I20260929 17:56:39.397583 17340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:56:39.395536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33709 (local address 127.0.98.3:59611)
0929 17:56:39.396046 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0929 17:56:39.396174 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":136,"threads_started":1}
W20260929 17:56:39.398079 17338 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:33709 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:33709: Client connection negotiation failed: client connection to 127.0.98.62:33709: connect: Connection refused (error 111)
I20260929 17:56:39.398787 17340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:56:39.398554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33709 (local address 127.0.98.3:34261)
0929 17:56:39.398675 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0929 17:56:39.398728 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260929 17:56:39.399389 17340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:56:39.399183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33709 (local address 127.0.98.3:51709)
0929 17:56:39.399301 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0929 17:56:39.399341 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260929 17:56:39.399559 17338 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260929 17:56:39.399596 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskup3Soe/build/debug/bin/kudu as pid 17209
I20260929 17:56:40.061388 17342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:56:40.060823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33709 (local address 127.0.98.1:48535)
0929 17:56:40.061200 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0929 17:56:40.061271 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":101,"threads_started":1}
I20260929 17:56:40.205797 17343 raft_consensus.cc:482] T 91ac91019d494f8a9d8d9024b1de49b7 P 433a7f95a3694411b43c7a77d7fc93ea [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 17:56:40.206012 17343 raft_consensus.cc:504] T 91ac91019d494f8a9d8d9024b1de49b7 P 433a7f95a3694411b43c7a77d7fc93ea [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8a734f480b34296ac1bb50572558f10" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34933 } } peers { permanent_uuid: "433a7f95a3694411b43c7a77d7fc93ea" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46395 } } peers { permanent_uuid: "b499dd73acdb48028230a6e4f02bd2d4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33451 } }
I20260929 17:56:40.207178 17343 leader_election.cc:288] T 91ac91019d494f8a9d8d9024b1de49b7 P 433a7f95a3694411b43c7a77d7fc93ea [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e8a734f480b34296ac1bb50572558f10 (127.0.98.3:34933), b499dd73acdb48028230a6e4f02bd2d4 (127.0.98.2:33451)
I20260929 17:56:40.213279 17347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:56:40.212810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33709 (local address 127.0.98.2:47285)
0929 17:56:40.213089 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0929 17:56:40.213157 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":90,"threads_started":1}
Sep 29 17:56:40 dist-test-slave-53nh krb5kdc[15758](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704598, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
Sep 29 17:56:40 dist-test-slave-53nh krb5kdc[15758](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704598, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
I20260929 17:56:40.221906 17342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:56:40.207376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34933 (local address 127.0.98.1:41929)
0929 17:56:40.207549 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0929 17:56:40.207564 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 17:56:40.207677 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:56:40.209704 (+ 2027us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:56:40.209711 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:56:40.210338 (+ 627us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:56:40.210779 (+ 441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:56:40.210796 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:56:40.212068 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:56:40.212071 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:56:40.213105 (+ 1034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:56:40.213115 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:56:40.214098 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:56:40.214102 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:56:40.214303 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:56:40.215010 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:56:40.215022 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:56:40.216823 (+ 1801us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:56:40.219027 (+ 2204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:56:40.219034 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:56:40.219048 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:56:40.219373 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:56:40.219786 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:56:40.219788 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:56:40.219790 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:56:40.219967 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:56:40.220698 (+ 731us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:56:40.220705 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:56:40.221099 (+ 394us) client_negotiation.cc:769] Sending connection context
0929 17:56:40.221309 (+ 210us) client_negotiation.cc:240] Negotiation successful
0929 17:56:40.221768 (+ 459us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95,"mutex_wait_us":278,"spinlock_wait_cycles":58496}
I20260929 17:56:40.221952 17344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:56:40.207523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41929 (local address 127.0.98.3:34933)
0929 17:56:40.207912 (+ 389us) server_negotiation.cc:206] Beginning negotiation
0929 17:56:40.207926 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 17:56:40.207971 (+ 45us) server_negotiation.cc:407] Connection header received
0929 17:56:40.208798 (+ 827us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:56:40.208806 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:56:40.209289 (+ 483us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:56:40.209610 (+ 321us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:56:40.210959 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:56:40.211922 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:56:40.213232 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:56:40.213975 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:56:40.214156 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:56:40.217118 (+ 2962us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:56:40.217125 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:56:40.217143 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:56:40.217177 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:56:40.218898 (+ 1721us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:56:40.219481 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:56:40.219486 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:56:40.219490 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:56:40.219636 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:56:40.220055 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:56:40.220058 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:56:40.220060 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:56:40.220575 (+ 515us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:56:40.220746 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0929 17:56:40.221477 (+ 731us) server_negotiation.cc:299] Negotiation successful
0929 17:56:40.221779 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":287,"thread_start_us":85,"threads_started":1}
I20260929 17:56:40.221905 17345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:56:40.207666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33451 (local address 127.0.98.1:45671)
0929 17:56:40.208054 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0929 17:56:40.208081 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0929 17:56:40.208180 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:56:40.209704 (+ 1524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:56:40.209711 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:56:40.210297 (+ 586us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:56:40.210778 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:56:40.210795 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:56:40.212073 (+ 1278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:56:40.212075 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:56:40.213115 (+ 1040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:56:40.213125 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:56:40.214135 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:56:40.214140 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:56:40.214284 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:56:40.214970 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:56:40.214990 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:56:40.217303 (+ 2313us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:56:40.219289 (+ 1986us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:56:40.219292 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:56:40.219294 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:56:40.219499 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:56:40.219838 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:56:40.219841 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:56:40.219843 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:56:40.219967 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:56:40.220698 (+ 731us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:56:40.220705 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:56:40.221099 (+ 394us) client_negotiation.cc:769] Sending connection context
0929 17:56:40.221309 (+ 210us) client_negotiation.cc:240] Negotiation successful
0929 17:56:40.221698 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":322,"mutex_wait_us":112,"spinlock_wait_cycles":14464,"thread_start_us":68,"threads_started":1}
I20260929 17:56:40.221952 17346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:56:40.208044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45671 (local address 127.0.98.2:33451)
0929 17:56:40.208394 (+ 350us) server_negotiation.cc:206] Beginning negotiation
0929 17:56:40.208404 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 17:56:40.208432 (+ 28us) server_negotiation.cc:407] Connection header received
0929 17:56:40.209047 (+ 615us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:56:40.209056 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:56:40.209355 (+ 299us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:56:40.210218 (+ 863us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:56:40.210945 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:56:40.211918 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:56:40.213249 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:56:40.213969 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:56:40.214513 (+ 544us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:56:40.217482 (+ 2969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:56:40.217489 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:56:40.217503 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:56:40.217536 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:56:40.219179 (+ 1643us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:56:40.219610 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:56:40.219615 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:56:40.219619 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:56:40.219756 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:56:40.220055 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:56:40.220058 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:56:40.220059 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:56:40.220575 (+ 516us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:56:40.220712 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0929 17:56:40.221477 (+ 765us) server_negotiation.cc:299] Negotiation successful
0929 17:56:40.221770 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":263,"thread_start_us":77,"threads_started":1}
W20260929 17:56:40.224448 16958 leader_election.cc:341] T 91ac91019d494f8a9d8d9024b1de49b7 P 433a7f95a3694411b43c7a77d7fc93ea [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e8a734f480b34296ac1bb50572558f10 (127.0.98.3:34933): Not found: Tablet not found: 91ac91019d494f8a9d8d9024b1de49b7
W20260929 17:56:40.224666 16958 leader_election.cc:341] T 91ac91019d494f8a9d8d9024b1de49b7 P 433a7f95a3694411b43c7a77d7fc93ea [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b499dd73acdb48028230a6e4f02bd2d4 (127.0.98.2:33451): Not found: Tablet not found: 91ac91019d494f8a9d8d9024b1de49b7
I20260929 17:56:40.224746 16958 leader_election.cc:302] T 91ac91019d494f8a9d8d9024b1de49b7 P 433a7f95a3694411b43c7a77d7fc93ea [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 433a7f95a3694411b43c7a77d7fc93ea; no voters: b499dd73acdb48028230a6e4f02bd2d4, e8a734f480b34296ac1bb50572558f10
I20260929 17:56:40.224985 17343 raft_consensus.cc:2726] T 91ac91019d494f8a9d8d9024b1de49b7 P 433a7f95a3694411b43c7a77d7fc93ea [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 17:56:40.400628 17348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:56:40.400165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33709 (local address 127.0.98.3:60263)
0929 17:56:40.400467 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0929 17:56:40.400530 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":104,"threads_started":1}
I20260929 17:56:41.062662 17349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:56:41.062226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33709 (local address 127.0.98.1:56825)
0929 17:56:41.062502 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0929 17:56:41.062564 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":79,"threads_started":1}
I20260929 17:56:41.215423 17350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:56:41.214958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33709 (local address 127.0.98.2:56003)
0929 17:56:41.215242 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0929 17:56:41.215305 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":78,"threads_started":1}
I20260929 17:56:41.401904 17351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:56:41.401437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33709 (local address 127.0.98.3:55059)
0929 17:56:41.401720 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0929 17:56:41.401787 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":76,"threads_started":1}
I20260929 17:56:41.812181 17352 raft_consensus.cc:482] T 91ac91019d494f8a9d8d9024b1de49b7 P 433a7f95a3694411b43c7a77d7fc93ea [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 17:56:41.812316 17352 raft_consensus.cc:504] T 91ac91019d494f8a9d8d9024b1de49b7 P 433a7f95a3694411b43c7a77d7fc93ea [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8a734f480b34296ac1bb50572558f10" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34933 } } peers { permanent_uuid: "433a7f95a3694411b43c7a77d7fc93ea" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46395 } } peers { permanent_uuid: "b499dd73acdb48028230a6e4f02bd2d4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33451 } }
I20260929 17:56:41.812726 17352 leader_election.cc:288] T 91ac91019d494f8a9d8d9024b1de49b7 P 433a7f95a3694411b43c7a77d7fc93ea [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e8a734f480b34296ac1bb50572558f10 (127.0.98.3:34933), b499dd73acdb48028230a6e4f02bd2d4 (127.0.98.2:33451)
W20260929 17:56:41.813621 16958 leader_election.cc:341] T 91ac91019d494f8a9d8d9024b1de49b7 P 433a7f95a3694411b43c7a77d7fc93ea [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e8a734f480b34296ac1bb50572558f10 (127.0.98.3:34933): Not found: Tablet not found: 91ac91019d494f8a9d8d9024b1de49b7
W20260929 17:56:41.813763 16958 leader_election.cc:341] T 91ac91019d494f8a9d8d9024b1de49b7 P 433a7f95a3694411b43c7a77d7fc93ea [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b499dd73acdb48028230a6e4f02bd2d4 (127.0.98.2:33451): Not found: Tablet not found: 91ac91019d494f8a9d8d9024b1de49b7
I20260929 17:56:41.813818 16958 leader_election.cc:302] T 91ac91019d494f8a9d8d9024b1de49b7 P 433a7f95a3694411b43c7a77d7fc93ea [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 433a7f95a3694411b43c7a77d7fc93ea; no voters: b499dd73acdb48028230a6e4f02bd2d4, e8a734f480b34296ac1bb50572558f10
I20260929 17:56:41.814004 17352 raft_consensus.cc:2726] T 91ac91019d494f8a9d8d9024b1de49b7 P 433a7f95a3694411b43c7a77d7fc93ea [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 17:56:42.064066 17353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:56:42.063515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33709 (local address 127.0.98.1:49137)
0929 17:56:42.063882 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0929 17:56:42.063965 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":125,"threads_started":1}
I20260929 17:56:42.216771 17354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:56:42.216257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33709 (local address 127.0.98.2:43053)
0929 17:56:42.216576 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0929 17:56:42.216644 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":87,"threads_started":1}
I20260929 17:56:42.403164 17355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:56:42.402726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33709 (local address 127.0.98.3:59285)
0929 17:56:42.403009 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0929 17:56:42.403069 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":99,"threads_started":1}
I20260929 17:56:43.065249 17356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:56:43.064793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33709 (local address 127.0.98.1:46303)
0929 17:56:43.065069 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0929 17:56:43.065134 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":76,"threads_started":1}
I20260929 17:56:43.218055 17357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:56:43.217642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33709 (local address 127.0.98.2:52041)
0929 17:56:43.217896 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0929 17:56:43.217956 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":78,"threads_started":1}
I20260929 17:56:43.404433 17358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:56:43.403958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33709 (local address 127.0.98.3:60773)
0929 17:56:43.404271 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0929 17:56:43.404336 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":83,"threads_started":1}
I20260929 17:56:43.843953 17359 raft_consensus.cc:482] T 91ac91019d494f8a9d8d9024b1de49b7 P 433a7f95a3694411b43c7a77d7fc93ea [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 17:56:43.844094 17359 raft_consensus.cc:504] T 91ac91019d494f8a9d8d9024b1de49b7 P 433a7f95a3694411b43c7a77d7fc93ea [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8a734f480b34296ac1bb50572558f10" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34933 } } peers { permanent_uuid: "433a7f95a3694411b43c7a77d7fc93ea" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46395 } } peers { permanent_uuid: "b499dd73acdb48028230a6e4f02bd2d4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33451 } }
I20260929 17:56:43.844519 17359 leader_election.cc:288] T 91ac91019d494f8a9d8d9024b1de49b7 P 433a7f95a3694411b43c7a77d7fc93ea [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e8a734f480b34296ac1bb50572558f10 (127.0.98.3:34933), b499dd73acdb48028230a6e4f02bd2d4 (127.0.98.2:33451)
W20260929 17:56:43.845448 16958 leader_election.cc:341] T 91ac91019d494f8a9d8d9024b1de49b7 P 433a7f95a3694411b43c7a77d7fc93ea [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b499dd73acdb48028230a6e4f02bd2d4 (127.0.98.2:33451): Not found: Tablet not found: 91ac91019d494f8a9d8d9024b1de49b7
W20260929 17:56:43.845590 16958 leader_election.cc:341] T 91ac91019d494f8a9d8d9024b1de49b7 P 433a7f95a3694411b43c7a77d7fc93ea [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e8a734f480b34296ac1bb50572558f10 (127.0.98.3:34933): Not found: Tablet not found: 91ac91019d494f8a9d8d9024b1de49b7
I20260929 17:56:43.845639 16958 leader_election.cc:302] T 91ac91019d494f8a9d8d9024b1de49b7 P 433a7f95a3694411b43c7a77d7fc93ea [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 433a7f95a3694411b43c7a77d7fc93ea; no voters: b499dd73acdb48028230a6e4f02bd2d4, e8a734f480b34296ac1bb50572558f10
I20260929 17:56:43.845806 17359 raft_consensus.cc:2726] T 91ac91019d494f8a9d8d9024b1de49b7 P 433a7f95a3694411b43c7a77d7fc93ea [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 17:56:44.066471 17360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:56:44.066045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33709 (local address 127.0.98.1:40607)
0929 17:56:44.066318 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0929 17:56:44.066372 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":71,"threads_started":1}
I20260929 17:56:44.219406 17361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:56:44.218935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33709 (local address 127.0.98.2:52999)
0929 17:56:44.219216 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0929 17:56:44.219281 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":88,"threads_started":1}
I20260929 17:56:44.405766 17362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:56:44.405328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33709 (local address 127.0.98.3:58047)
0929 17:56:44.405590 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0929 17:56:44.405651 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":74,"threads_started":1}
I20260929 17:56:45.067615 17363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:56:45.067162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33709 (local address 127.0.98.1:44779)
0929 17:56:45.067459 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0929 17:56:45.067520 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":79,"threads_started":1}
I20260929 17:56:45.220518 17364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:56:45.220093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33709 (local address 127.0.98.2:56659)
0929 17:56:45.220364 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0929 17:56:45.220420 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":89,"threads_started":1}
I20260929 17:56:45.406839 17365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:56:45.406450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33709 (local address 127.0.98.3:59515)
0929 17:56:45.406676 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0929 17:56:45.406733 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":70,"threads_started":1}
I20260929 17:56:46.068807 17366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:56:46.068376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33709 (local address 127.0.98.1:58257)
0929 17:56:46.068627 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0929 17:56:46.068690 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":77,"threads_started":1}
I20260929 17:56:46.221833 17367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:56:46.221350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33709 (local address 127.0.98.2:47851)
0929 17:56:46.221664 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0929 17:56:46.221728 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":85,"threads_started":1}
I20260929 17:56:46.408106 17368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:56:46.407615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33709 (local address 127.0.98.3:49333)
0929 17:56:46.407944 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0929 17:56:46.408008 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":74,"threads_started":1}
I20260929 17:56:47.070163 17369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:56:47.069721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33709 (local address 127.0.98.1:49471)
0929 17:56:47.069997 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0929 17:56:47.070064 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":79,"threads_started":1}
I20260929 17:56:47.223021 17370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:56:47.222588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33709 (local address 127.0.98.2:48121)
0929 17:56:47.222849 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0929 17:56:47.222920 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":76,"threads_started":1}
I20260929 17:56:47.409534 17371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:56:47.409100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33709 (local address 127.0.98.3:54607)
0929 17:56:47.409362 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0929 17:56:47.409430 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":74,"threads_started":1}
I20260929 17:56:48.071399 17372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:56:48.070962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33709 (local address 127.0.98.1:36837)
0929 17:56:48.071222 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0929 17:56:48.071285 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":81,"threads_started":1}
I20260929 17:56:48.224195 17373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:56:48.223738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33709 (local address 127.0.98.2:38719)
0929 17:56:48.224028 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0929 17:56:48.224092 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":90,"threads_started":1}
I20260929 17:56:48.410797 17374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:56:48.410357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33709 (local address 127.0.98.3:37273)
0929 17:56:48.410633 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0929 17:56:48.410699 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":77,"threads_started":1}
I20260929 17:56:48.843359 17375 raft_consensus.cc:482] T 91ac91019d494f8a9d8d9024b1de49b7 P 433a7f95a3694411b43c7a77d7fc93ea [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 17:56:48.843502 17375 raft_consensus.cc:504] T 91ac91019d494f8a9d8d9024b1de49b7 P 433a7f95a3694411b43c7a77d7fc93ea [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8a734f480b34296ac1bb50572558f10" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34933 } } peers { permanent_uuid: "433a7f95a3694411b43c7a77d7fc93ea" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46395 } } peers { permanent_uuid: "b499dd73acdb48028230a6e4f02bd2d4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33451 } }
I20260929 17:56:48.843986 17375 leader_election.cc:288] T 91ac91019d494f8a9d8d9024b1de49b7 P 433a7f95a3694411b43c7a77d7fc93ea [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e8a734f480b34296ac1bb50572558f10 (127.0.98.3:34933), b499dd73acdb48028230a6e4f02bd2d4 (127.0.98.2:33451)
W20260929 17:56:48.844882 16958 leader_election.cc:341] T 91ac91019d494f8a9d8d9024b1de49b7 P 433a7f95a3694411b43c7a77d7fc93ea [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e8a734f480b34296ac1bb50572558f10 (127.0.98.3:34933): Not found: Tablet not found: 91ac91019d494f8a9d8d9024b1de49b7
W20260929 17:56:48.845036 16958 leader_election.cc:341] T 91ac91019d494f8a9d8d9024b1de49b7 P 433a7f95a3694411b43c7a77d7fc93ea [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b499dd73acdb48028230a6e4f02bd2d4 (127.0.98.2:33451): Not found: Tablet not found: 91ac91019d494f8a9d8d9024b1de49b7
I20260929 17:56:48.845088 16958 leader_election.cc:302] T 91ac91019d494f8a9d8d9024b1de49b7 P 433a7f95a3694411b43c7a77d7fc93ea [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 433a7f95a3694411b43c7a77d7fc93ea; no voters: b499dd73acdb48028230a6e4f02bd2d4, e8a734f480b34296ac1bb50572558f10
I20260929 17:56:48.845256 17375 raft_consensus.cc:2726] T 91ac91019d494f8a9d8d9024b1de49b7 P 433a7f95a3694411b43c7a77d7fc93ea [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 17:56:49.072773 17376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:56:49.072334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33709 (local address 127.0.98.1:40479)
0929 17:56:49.072607 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0929 17:56:49.072674 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":98,"threads_started":1}
I20260929 17:56:49.225508 17377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:56:49.225076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33709 (local address 127.0.98.2:57851)
0929 17:56:49.225346 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0929 17:56:49.225411 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":96,"threads_started":1}
I20260929 17:56:49.412101 17378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:56:49.411616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33709 (local address 127.0.98.3:46481)
0929 17:56:49.411939 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0929 17:56:49.412001 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":126,"threads_started":1}
I20260929 17:56:50.073978 17379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:56:50.073548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33709 (local address 127.0.98.1:39749)
0929 17:56:50.073820 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0929 17:56:50.073883 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":77,"threads_started":1}
I20260929 17:56:50.226642 17380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:56:50.226221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33709 (local address 127.0.98.2:57617)
0929 17:56:50.226485 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0929 17:56:50.226542 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":78,"threads_started":1}
I20260929 17:56:50.413422 17381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:56:50.412890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33709 (local address 127.0.98.3:56179)
0929 17:56:50.413224 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0929 17:56:50.413296 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":120,"threads_started":1}
I20260929 17:56:51.075145 17382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:56:51.074718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33709 (local address 127.0.98.1:56325)
0929 17:56:51.074987 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0929 17:56:51.075047 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":74,"threads_started":1}
I20260929 17:56:51.227836 17383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:56:51.227439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33709 (local address 127.0.98.2:42773)
0929 17:56:51.227681 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0929 17:56:51.227740 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":71,"threads_started":1}
I20260929 17:56:51.414685 17384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:56:51.414217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33709 (local address 127.0.98.3:40557)
0929 17:56:51.414484 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0929 17:56:51.414548 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":81,"threads_started":1}
I20260929 17:56:51.631309 17385 raft_consensus.cc:482] T 91ac91019d494f8a9d8d9024b1de49b7 P 433a7f95a3694411b43c7a77d7fc93ea [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 17:56:51.631456 17385 raft_consensus.cc:504] T 91ac91019d494f8a9d8d9024b1de49b7 P 433a7f95a3694411b43c7a77d7fc93ea [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8a734f480b34296ac1bb50572558f10" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34933 } } peers { permanent_uuid: "433a7f95a3694411b43c7a77d7fc93ea" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46395 } } peers { permanent_uuid: "b499dd73acdb48028230a6e4f02bd2d4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33451 } }
I20260929 17:56:51.631848 17385 leader_election.cc:288] T 91ac91019d494f8a9d8d9024b1de49b7 P 433a7f95a3694411b43c7a77d7fc93ea [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e8a734f480b34296ac1bb50572558f10 (127.0.98.3:34933), b499dd73acdb48028230a6e4f02bd2d4 (127.0.98.2:33451)
W20260929 17:56:51.632712 16958 leader_election.cc:341] T 91ac91019d494f8a9d8d9024b1de49b7 P 433a7f95a3694411b43c7a77d7fc93ea [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e8a734f480b34296ac1bb50572558f10 (127.0.98.3:34933): Not found: Tablet not found: 91ac91019d494f8a9d8d9024b1de49b7
W20260929 17:56:51.632877 16958 leader_election.cc:341] T 91ac91019d494f8a9d8d9024b1de49b7 P 433a7f95a3694411b43c7a77d7fc93ea [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b499dd73acdb48028230a6e4f02bd2d4 (127.0.98.2:33451): Not found: Tablet not found: 91ac91019d494f8a9d8d9024b1de49b7
I20260929 17:56:51.632942 16958 leader_election.cc:302] T 91ac91019d494f8a9d8d9024b1de49b7 P 433a7f95a3694411b43c7a77d7fc93ea [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 433a7f95a3694411b43c7a77d7fc93ea; no voters: b499dd73acdb48028230a6e4f02bd2d4, e8a734f480b34296ac1bb50572558f10
I20260929 17:56:51.633123 17385 raft_consensus.cc:2726] T 91ac91019d494f8a9d8d9024b1de49b7 P 433a7f95a3694411b43c7a77d7fc93ea [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 17:56:52.076450 17386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:56:52.075980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33709 (local address 127.0.98.1:56371)
0929 17:56:52.076279 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0929 17:56:52.076345 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":88,"threads_started":1}
I20260929 17:56:52.229193 17387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:56:52.228653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33709 (local address 127.0.98.2:60261)
0929 17:56:52.228976 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0929 17:56:52.229091 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":81,"threads_started":1}
I20260929 17:56:52.415974 17388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:56:52.415466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33709 (local address 127.0.98.3:35449)
0929 17:56:52.415771 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0929 17:56:52.415877 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":90,"threads_started":1}
I20260929 17:56:53.077754 17389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:56:53.077295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33709 (local address 127.0.98.1:52671)
0929 17:56:53.077594 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0929 17:56:53.077658 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":94,"threads_started":1}
I20260929 17:56:53.230437 17390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:56:53.230006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33709 (local address 127.0.98.2:57115)
0929 17:56:53.230277 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0929 17:56:53.230338 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":76,"threads_started":1}
I20260929 17:56:53.417371 17391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:56:53.416813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33709 (local address 127.0.98.3:41473)
0929 17:56:53.417194 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0929 17:56:53.417268 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":79,"threads_started":1}
I20260929 17:56:54.079386 17392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:56:54.078883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33709 (local address 127.0.98.1:53071)
0929 17:56:54.079181 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0929 17:56:54.079256 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":83,"threads_started":1}
I20260929 17:56:54.231824 17393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:56:54.231373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33709 (local address 127.0.98.2:43563)
0929 17:56:54.231658 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0929 17:56:54.231723 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":81,"threads_started":1}
I20260929 17:56:54.405582 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskup3Soe/build/debug/bin/kudu with pid 16902
I20260929 17:56:54.410940 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskup3Soe/build/debug/bin/kudu with pid 17075
I20260929 17:56:54.415915 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskup3Soe/build/debug/bin/kudu with pid 17209
2026-09-29T17:56:54Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (58838 ms)
[----------] 30 tests from SecurityITest (238432 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704375975685-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 17:56:54 dist-test-slave-53nh krb5kdc[17396](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 17:56:54 dist-test-slave-53nh krb5kdc[17396](info): set up 2 sockets
Sep 29 17:56:54 dist-test-slave-53nh krb5kdc[17396](info): commencing operation
krb5kdc: starting...
W20260929 17:56:55.368659 392 mini_kdc.cc:121] Time spent starting KDC: real 0.932s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704375975685-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704375975685-392-0/krb5kdc/test-user.keytab.
Sep 29 17:56:55 dist-test-slave-53nh krb5kdc[17396](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704615, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T17:56:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T17:56:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704375975685-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704375975685-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 17:56:55.488606 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskup3Soe/build/debug/bin/kudu
/tmp/dist-test-taskup3Soe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704375975685-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704375975685-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704375975685-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704375975685-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33981
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704375975685-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45387
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33981
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704375975685-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704375975685-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704375975685-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:56:55.601351 17409 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:56:55.601637 17409 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:56:55.601687 17409 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:56:55.605834 17409 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 17:56:55.605912 17409 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:56:55.605931 17409 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:56:55.605947 17409 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 17:56:55.605962 17409 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 17:56:55.611506 17409 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45387
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704375975685-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704375975685-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33981
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704375975685-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33981
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704375975685-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17409
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704375975685-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:56:55.612670 17409 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:56:55.613801 17409 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 17:56:55.622216 17409 server_base.cc:1034] running on GCE node
W20260929 17:56:55.622254 17414 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:56:55.622254 17415 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:56:55.622488 17417 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:56:55.623005 17409 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:56:55.624063 17409 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:56:55.625219 17409 hybrid_clock.cc:648] HybridClock initialized: now 1790704615625202 us; error 32 us; skew 500 ppm
Sep 29 17:56:55 dist-test-slave-53nh krb5kdc[17396](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704615, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:56:55.627878 17409 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 17:56:55.628793 17409 webserver.cc:460] Webserver started at http://127.0.98.62:34637/ using document root <none> and password file <none>
I20260929 17:56:55.629388 17409 fs_manager.cc:359] Metadata directory not provided
I20260929 17:56:55.629463 17409 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:56:55.629679 17409 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:56:55.631678 17409 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704375975685-392-0/minicluster-data/master-0/data/instance:
uuid: "a40851c12f464ec996f46ef7a79687c6"
format_stamp: "Formatted at 2026-09-29 17:56:55 on dist-test-slave-53nh"
server_key: "58d96497a7521e6eace9fe5e684319c1"
server_key_iv: "5bf03777d7e0efbf655deddef2b0f6d1"
server_key_version: "encryptionkey@0"
I20260929 17:56:55.632227 17409 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704375975685-392-0/minicluster-data/master-0/wal/instance:
uuid: "a40851c12f464ec996f46ef7a79687c6"
format_stamp: "Formatted at 2026-09-29 17:56:55 on dist-test-slave-53nh"
server_key: "58d96497a7521e6eace9fe5e684319c1"
server_key_iv: "5bf03777d7e0efbf655deddef2b0f6d1"
server_key_version: "encryptionkey@0"
I20260929 17:56:55.635954 17409 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260929 17:56:55.638350 17424 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:56:55.639662 17409 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 17:56:55.639786 17409 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704375975685-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704375975685-392-0/minicluster-data/master-0/wal
uuid: "a40851c12f464ec996f46ef7a79687c6"
format_stamp: "Formatted at 2026-09-29 17:56:55 on dist-test-slave-53nh"
server_key: "58d96497a7521e6eace9fe5e684319c1"
server_key_iv: "5bf03777d7e0efbf655deddef2b0f6d1"
server_key_version: "encryptionkey@0"
I20260929 17:56:55.639889 17409 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704375975685-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704375975685-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704375975685-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:56:55.657779 17409 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:56:55.660889 17409 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:56:55.661092 17409 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:56:55.667809 17409 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33981
I20260929 17:56:55.667843 17476 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33981 every 8 connection(s)
I20260929 17:56:55.668756 17409 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704375975685-392-0/minicluster-data/master-0/data/info.pb
I20260929 17:56:55.671581 17477 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:56:55.674971 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskup3Soe/build/debug/bin/kudu as pid 17409
I20260929 17:56:55.675092 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704375975685-392-0/minicluster-data/master-0/wal/instance
I20260929 17:56:55.675382 392 external_mini_cluster.cc:1428] Setting key 72f34ebd8d78344486c3d474426933eb
I20260929 17:56:55.678093 17477 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a40851c12f464ec996f46ef7a79687c6: Bootstrap starting.
I20260929 17:56:55.680956 17477 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a40851c12f464ec996f46ef7a79687c6: Neither blocks nor log segments found. Creating new log.
I20260929 17:56:55.681849 17477 log.cc:824] T 00000000000000000000000000000000 P a40851c12f464ec996f46ef7a79687c6: Log is configured to *not* fsync() on all Append() calls
I20260929 17:56:55.683804 17477 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a40851c12f464ec996f46ef7a79687c6: No bootstrap required, opened a new log
Sep 29 17:56:55 dist-test-slave-53nh krb5kdc[17396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704615, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 17:56:55.687309 17477 raft_consensus.cc:348] T 00000000000000000000000000000000 P a40851c12f464ec996f46ef7a79687c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a40851c12f464ec996f46ef7a79687c6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33981 } }
I20260929 17:56:55.687572 17477 raft_consensus.cc:374] T 00000000000000000000000000000000 P a40851c12f464ec996f46ef7a79687c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:56:55.687646 17477 raft_consensus.cc:729] T 00000000000000000000000000000000 P a40851c12f464ec996f46ef7a79687c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a40851c12f464ec996f46ef7a79687c6, State: Initialized, Role: FOLLOWER
I20260929 17:56:55.688133 17477 consensus_queue.cc:260] T 00000000000000000000000000000000 P a40851c12f464ec996f46ef7a79687c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a40851c12f464ec996f46ef7a79687c6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33981 } }
I20260929 17:56:55.688278 17477 raft_consensus.cc:388] T 00000000000000000000000000000000 P a40851c12f464ec996f46ef7a79687c6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 17:56:55.688369 17477 raft_consensus.cc:482] T 00000000000000000000000000000000 P a40851c12f464ec996f46ef7a79687c6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 17:56:55.688488 17477 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a40851c12f464ec996f46ef7a79687c6 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:56:55.689682 17477 raft_consensus.cc:504] T 00000000000000000000000000000000 P a40851c12f464ec996f46ef7a79687c6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a40851c12f464ec996f46ef7a79687c6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33981 } }
I20260929 17:56:55.690119 17477 leader_election.cc:302] T 00000000000000000000000000000000 P a40851c12f464ec996f46ef7a79687c6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a40851c12f464ec996f46ef7a79687c6; no voters:
I20260929 17:56:55.690443 17477 leader_election.cc:288] T 00000000000000000000000000000000 P a40851c12f464ec996f46ef7a79687c6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 17:56:55.690548 17482 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a40851c12f464ec996f46ef7a79687c6 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 17:56:55.690789 17482 raft_consensus.cc:686] T 00000000000000000000000000000000 P a40851c12f464ec996f46ef7a79687c6 [term 1 LEADER]: Becoming Leader. State: Replica: a40851c12f464ec996f46ef7a79687c6, State: Running, Role: LEADER
I20260929 17:56:55.691151 17482 consensus_queue.cc:237] T 00000000000000000000000000000000 P a40851c12f464ec996f46ef7a79687c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a40851c12f464ec996f46ef7a79687c6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33981 } }
I20260929 17:56:55.691326 17480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:56:55.676835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60266 (local address 127.0.98.62:33981)
0929 17:56:55.677293 (+ 458us) server_negotiation.cc:206] Beginning negotiation
0929 17:56:55.677314 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0929 17:56:55.677358 (+ 44us) server_negotiation.cc:407] Connection header received
0929 17:56:55.678213 (+ 855us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:56:55.678239 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:56:55.678710 (+ 471us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:56:55.679104 (+ 394us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:56:55.679767 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:56:55.681269 (+ 1502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:56:55.682264 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:56:55.682750 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:56:55.682907 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:56:55.685361 (+ 2454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:56:55.685367 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:56:55.685416 (+ 49us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:56:55.685447 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:56:55.687900 (+ 2453us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:56:55.688333 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:56:55.688338 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:56:55.688343 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:56:55.688464 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:56:55.688807 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:56:55.688809 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:56:55.688810 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:56:55.689286 (+ 476us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:56:55.689440 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0929 17:56:55.689819 (+ 379us) server_negotiation.cc:299] Negotiation successful
0929 17:56:55.690224 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298,"thread_start_us":141,"threads_started":1}
I20260929 17:56:55.691730 17477 sys_catalog.cc:564] T 00000000000000000000000000000000 P a40851c12f464ec996f46ef7a79687c6 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 17:56:55.692425 17482 sys_catalog.cc:455] T 00000000000000000000000000000000 P a40851c12f464ec996f46ef7a79687c6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a40851c12f464ec996f46ef7a79687c6. Latest consensus state: current_term: 1 leader_uuid: "a40851c12f464ec996f46ef7a79687c6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a40851c12f464ec996f46ef7a79687c6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33981 } } }
I20260929 17:56:55.692633 17482 sys_catalog.cc:458] T 00000000000000000000000000000000 P a40851c12f464ec996f46ef7a79687c6 [sys.catalog]: This master's current role is: LEADER
I20260929 17:56:55.692849 17483 sys_catalog.cc:455] T 00000000000000000000000000000000 P a40851c12f464ec996f46ef7a79687c6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a40851c12f464ec996f46ef7a79687c6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a40851c12f464ec996f46ef7a79687c6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33981 } } }
I20260929 17:56:55.692965 17483 sys_catalog.cc:458] T 00000000000000000000000000000000 P a40851c12f464ec996f46ef7a79687c6 [sys.catalog]: This master's current role is: LEADER
I20260929 17:56:55.693856 17491 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 17:56:55.696870 17491 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 17:56:55.702155 17491 catalog_manager.cc:1347] Generated new cluster ID: 1b14951fbd81426eb729e0a8162de38b
I20260929 17:56:55.702221 17491 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 17:56:55.738679 17491 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 17:56:55.739711 17491 catalog_manager.cc:1504] Loading token signing keys...
I20260929 17:56:55.743994 17491 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a40851c12f464ec996f46ef7a79687c6: Generated new TSK 0
I20260929 17:56:55.744776 17491 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704375975685-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704375975685-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260929 17:56:55.784562 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskup3Soe/build/debug/bin/kudu
/tmp/dist-test-taskup3Soe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704375975685-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704375975685-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704375975685-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704375975685-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33981
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704375975685-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45387
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704375975685-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704375975685-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704375975685-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:56:55.908475 17503 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:56:55.908766 17503 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:56:55.908826 17503 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:56:55.913401 17503 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:56:55.913483 17503 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:56:55.913570 17503 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 17:56:55.919098 17503 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45387
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704375975685-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704375975685-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704375975685-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704375975685-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33981
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17503
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704375975685-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:56:55.920219 17503 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:56:55.921291 17503 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:56:55.931383 17511 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:56:55.931419 17508 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:56:55.931581 17503 server_base.cc:1034] running on GCE node
W20260929 17:56:55.931413 17509 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:56:55.932082 17503 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:56:55.932701 17503 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:56:55.933866 17503 hybrid_clock.cc:648] HybridClock initialized: now 1790704615933827 us; error 50 us; skew 500 ppm
Sep 29 17:56:55 dist-test-slave-53nh krb5kdc[17396](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704615, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:56:55.936571 17503 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260929 17:56:55.937492 17503 webserver.cc:460] Webserver started at http://127.0.98.1:45313/ using document root <none> and password file <none>
I20260929 17:56:55.938030 17503 fs_manager.cc:359] Metadata directory not provided
I20260929 17:56:55.938093 17503 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:56:55.938297 17503 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:56:55.940331 17503 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704375975685-392-0/minicluster-data/ts-0/data/instance:
uuid: "71aeb1d9d2a540d6807b222a31af7277"
format_stamp: "Formatted at 2026-09-29 17:56:55 on dist-test-slave-53nh"
server_key: "e6a0e3e7dc233f6ca906f3643ae089df"
server_key_iv: "8b1f3977c3cabfbcae39d7f257890a16"
server_key_version: "encryptionkey@0"
I20260929 17:56:55.940862 17503 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704375975685-392-0/minicluster-data/ts-0/wal/instance:
uuid: "71aeb1d9d2a540d6807b222a31af7277"
format_stamp: "Formatted at 2026-09-29 17:56:55 on dist-test-slave-53nh"
server_key: "e6a0e3e7dc233f6ca906f3643ae089df"
server_key_iv: "8b1f3977c3cabfbcae39d7f257890a16"
server_key_version: "encryptionkey@0"
I20260929 17:56:55.944662 17503 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.000s
I20260929 17:56:55.947283 17518 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:56:55.948467 17503 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 17:56:55.948575 17503 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704375975685-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704375975685-392-0/minicluster-data/ts-0/wal
uuid: "71aeb1d9d2a540d6807b222a31af7277"
format_stamp: "Formatted at 2026-09-29 17:56:55 on dist-test-slave-53nh"
server_key: "e6a0e3e7dc233f6ca906f3643ae089df"
server_key_iv: "8b1f3977c3cabfbcae39d7f257890a16"
server_key_version: "encryptionkey@0"
I20260929 17:56:55.948668 17503 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704375975685-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704375975685-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704375975685-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:56:55.958535 17503 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:56:55.961545 17503 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:56:55.961717 17503 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:56:55.962255 17503 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 17:56:55.963053 17503 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:56:55.963099 17503 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:56:55.963150 17503 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:56:55.963178 17503 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 17:56:55.971751 17503 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44585
I20260929 17:56:55.971784 17631 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44585 every 8 connection(s)
I20260929 17:56:55.972671 17503 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704375975685-392-0/minicluster-data/ts-0/data/info.pb
I20260929 17:56:55.981434 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskup3Soe/build/debug/bin/kudu as pid 17503
I20260929 17:56:55.981570 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704375975685-392-0/minicluster-data/ts-0/wal/instance
I20260929 17:56:55.981837 392 external_mini_cluster.cc:1428] Setting key cc8ac9cdf6091546832cd94e10caa3f5
Sep 29 17:56:55 dist-test-slave-53nh krb5kdc[17396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704615, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 17:56:55.986509 17480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:56:55.974704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54083 (local address 127.0.98.62:33981)
0929 17:56:55.974864 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0929 17:56:55.974869 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:56:55.975814 (+ 945us) server_negotiation.cc:407] Connection header received
0929 17:56:55.977158 (+ 1344us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:56:55.977161 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:56:55.977215 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:56:55.977297 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:56:55.978510 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:56:55.979043 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:56:55.980183 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:56:55.980424 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:56:55.980498 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:56:55.983041 (+ 2543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:56:55.983046 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:56:55.983048 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:56:55.983075 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:56:55.984462 (+ 1387us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:56:55.984987 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:56:55.984990 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:56:55.984992 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:56:55.985105 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:56:55.985425 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:56:55.985428 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:56:55.985430 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:56:55.985648 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:56:55.985745 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0929 17:56:55.986225 (+ 480us) server_negotiation.cc:299] Negotiation successful
0929 17:56:55.986352 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 17:56:55.987372 17634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:56:55.975146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33981 (local address 127.0.98.1:54083)
0929 17:56:55.975645 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0929 17:56:55.975691 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 17:56:55.976905 (+ 1214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:56:55.977466 (+ 561us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:56:55.977474 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:56:55.977842 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:56:55.978370 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:56:55.978384 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:56:55.979158 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:56:55.979161 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:56:55.980087 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:56:55.980097 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:56:55.980555 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:56:55.980558 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:56:55.980705 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:56:55.981375 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:56:55.981392 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:56:55.982873 (+ 1481us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:56:55.984584 (+ 1711us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:56:55.984591 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:56:55.984610 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:56:55.984902 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:56:55.985180 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:56:55.985182 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:56:55.985183 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:56:55.985345 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:56:55.985731 (+ 386us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:56:55.985736 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:56:55.985997 (+ 261us) client_negotiation.cc:769] Sending connection context
0929 17:56:55.986156 (+ 159us) client_negotiation.cc:240] Negotiation successful
0929 17:56:55.986449 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":135,"threads_started":1}
I20260929 17:56:55.988804 17632 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33981
I20260929 17:56:55.989046 17632 heartbeater.cc:464] Registering TS with master...
I20260929 17:56:55.989522 17632 heartbeater.cc:511] Master 127.0.98.62:33981 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260929 17:56:55.991191 17441 ts_manager.cc:194] Registered new tserver with Master: 71aeb1d9d2a540d6807b222a31af7277 (127.0.98.1:44585)
I20260929 17:56:55.992344 17441 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:54083
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704375975685-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704375975685-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260929 17:56:56.008844 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskup3Soe/build/debug/bin/kudu
/tmp/dist-test-taskup3Soe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704375975685-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704375975685-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704375975685-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704375975685-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33981
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704375975685-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45387
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704375975685-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704375975685-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704375975685-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:56:56.115985 17637 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:56:56.116272 17637 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:56:56.116317 17637 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:56:56.120334 17637 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:56:56.120393 17637 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:56:56.120455 17637 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 17:56:56.125792 17637 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45387
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704375975685-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704375975685-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704375975685-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704375975685-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33981
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17637
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704375975685-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:56:56.126927 17637 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:56:56.128010 17637 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 17:56:56.137532 17637 server_base.cc:1034] running on GCE node
W20260929 17:56:56.137517 17645 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:56:56.137480 17643 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:56:56.137480 17642 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:56:56.138118 17637 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:56:56.138777 17637 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:56:56.139954 17637 hybrid_clock.cc:648] HybridClock initialized: now 1790704616139935 us; error 30 us; skew 500 ppm
Sep 29 17:56:56 dist-test-slave-53nh krb5kdc[17396](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704616, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:56:56.142805 17637 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260929 17:56:56.143766 17637 webserver.cc:460] Webserver started at http://127.0.98.2:46155/ using document root <none> and password file <none>
I20260929 17:56:56.144343 17637 fs_manager.cc:359] Metadata directory not provided
I20260929 17:56:56.144409 17637 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:56:56.144621 17637 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:56:56.146772 17637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704375975685-392-0/minicluster-data/ts-1/data/instance:
uuid: "a9e1d9865cc1424387add2b266e9e2d0"
format_stamp: "Formatted at 2026-09-29 17:56:56 on dist-test-slave-53nh"
server_key: "7cd3201eb11006ab685e8381c16f2b4e"
server_key_iv: "9dde3d292d0e62f8fb7b23ff6c5c13ec"
server_key_version: "encryptionkey@0"
I20260929 17:56:56.147334 17637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704375975685-392-0/minicluster-data/ts-1/wal/instance:
uuid: "a9e1d9865cc1424387add2b266e9e2d0"
format_stamp: "Formatted at 2026-09-29 17:56:56 on dist-test-slave-53nh"
server_key: "7cd3201eb11006ab685e8381c16f2b4e"
server_key_iv: "9dde3d292d0e62f8fb7b23ff6c5c13ec"
server_key_version: "encryptionkey@0"
I20260929 17:56:56.151165 17637 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260929 17:56:56.153712 17652 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:56:56.154943 17637 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 17:56:56.155066 17637 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704375975685-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704375975685-392-0/minicluster-data/ts-1/wal
uuid: "a9e1d9865cc1424387add2b266e9e2d0"
format_stamp: "Formatted at 2026-09-29 17:56:56 on dist-test-slave-53nh"
server_key: "7cd3201eb11006ab685e8381c16f2b4e"
server_key_iv: "9dde3d292d0e62f8fb7b23ff6c5c13ec"
server_key_version: "encryptionkey@0"
I20260929 17:56:56.155164 17637 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704375975685-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704375975685-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704375975685-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:56:56.166147 17637 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:56:56.169526 17637 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:56:56.169713 17637 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:56:56.170325 17637 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 17:56:56.171154 17637 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:56:56.171200 17637 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:56:56.171262 17637 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:56:56.171288 17637 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 17:56:56.179756 17637 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37843
I20260929 17:56:56.179791 17765 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37843 every 8 connection(s)
I20260929 17:56:56.180655 17637 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704375975685-392-0/minicluster-data/ts-1/data/info.pb
I20260929 17:56:56.185222 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskup3Soe/build/debug/bin/kudu as pid 17637
I20260929 17:56:56.185355 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704375975685-392-0/minicluster-data/ts-1/wal/instance
I20260929 17:56:56.185662 392 external_mini_cluster.cc:1428] Setting key 56f90a349b3a2c814274a9abeb450164
Sep 29 17:56:56 dist-test-slave-53nh krb5kdc[17396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704616, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 17:56:56.195976 17480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:56:56.182558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35017 (local address 127.0.98.62:33981)
0929 17:56:56.182747 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0929 17:56:56.182752 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:56:56.183570 (+ 818us) server_negotiation.cc:407] Connection header received
0929 17:56:56.184898 (+ 1328us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:56:56.184902 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:56:56.184960 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:56:56.185063 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:56:56.186368 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:56:56.186985 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:56:56.188666 (+ 1681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:56:56.188921 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:56:56.189045 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:56:56.191930 (+ 2885us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:56:56.191934 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:56:56.191936 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:56:56.191962 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:56:56.193706 (+ 1744us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:56:56.194350 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:56:56.194353 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:56:56.194355 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:56:56.194432 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:56:56.194789 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:56:56.194791 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:56:56.194793 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:56:56.194972 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:56:56.195045 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0929 17:56:56.195696 (+ 651us) server_negotiation.cc:299] Negotiation successful
0929 17:56:56.195835 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260929 17:56:56.196957 17768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:56:56.182922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33981 (local address 127.0.98.2:35017)
0929 17:56:56.183416 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0929 17:56:56.183461 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 17:56:56.184681 (+ 1220us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:56:56.185247 (+ 566us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:56:56.185254 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:56:56.185656 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:56:56.186191 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:56:56.186217 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:56:56.187126 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:56:56.187130 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:56:56.188522 (+ 1392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:56:56.188533 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:56:56.189094 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:56:56.189108 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:56:56.189320 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:56:56.189950 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:56:56.189972 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:56:56.191696 (+ 1724us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:56:56.193849 (+ 2153us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:56:56.193856 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:56:56.193874 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:56:56.194249 (+ 375us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:56:56.194522 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:56:56.194526 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:56:56.194528 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:56:56.194707 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:56:56.195070 (+ 363us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:56:56.195076 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:56:56.195427 (+ 351us) client_negotiation.cc:769] Sending connection context
0929 17:56:56.195655 (+ 228us) client_negotiation.cc:240] Negotiation successful
0929 17:56:56.196017 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":122,"threads_started":1}
I20260929 17:56:56.198158 17766 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33981
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260929 17:56:56.198449 17766 heartbeater.cc:464] Registering TS with master...
I20260929 17:56:56.198972 17766 heartbeater.cc:511] Master 127.0.98.62:33981 requested a full tablet report, sending...
I20260929 17:56:56.199982 17441 ts_manager.cc:194] Registered new tserver with Master: a9e1d9865cc1424387add2b266e9e2d0 (127.0.98.2:37843)
I20260929 17:56:56.200547 17441 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:35017
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704375975685-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704375975685-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260929 17:56:56.213680 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskup3Soe/build/debug/bin/kudu
/tmp/dist-test-taskup3Soe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704375975685-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704375975685-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704375975685-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704375975685-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33981
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704375975685-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45387
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704375975685-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704375975685-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704375975685-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:56:56.325296 17771 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:56:56.325610 17771 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:56:56.325654 17771 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:56:56.329913 17771 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:56:56.329985 17771 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:56:56.330049 17771 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 17:56:56.335739 17771 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45387
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704375975685-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704375975685-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704375975685-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704375975685-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33981
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17771
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704375975685-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:56:56.336833 17771 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:56:56.337896 17771 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:56:56.347656 17776 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:56:56.347698 17771 server_base.cc:1034] running on GCE node
W20260929 17:56:56.347645 17777 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:56:56.347673 17779 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:56:56.348443 17771 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:56:56.349109 17771 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:56:56.350325 17771 hybrid_clock.cc:648] HybridClock initialized: now 1790704616350305 us; error 28 us; skew 500 ppm
Sep 29 17:56:56 dist-test-slave-53nh krb5kdc[17396](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704616, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:56:56.353094 17771 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260929 17:56:56.353960 17771 webserver.cc:460] Webserver started at http://127.0.98.3:38103/ using document root <none> and password file <none>
I20260929 17:56:56.354508 17771 fs_manager.cc:359] Metadata directory not provided
I20260929 17:56:56.354564 17771 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:56:56.354725 17771 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:56:56.356757 17771 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704375975685-392-0/minicluster-data/ts-2/data/instance:
uuid: "40d6a12e22734200b84df12f2fe0c9c2"
format_stamp: "Formatted at 2026-09-29 17:56:56 on dist-test-slave-53nh"
server_key: "e72e4d151a197509acb9ec2736113736"
server_key_iv: "1eb50673916778f87d77805d8ce6f9a1"
server_key_version: "encryptionkey@0"
I20260929 17:56:56.357301 17771 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704375975685-392-0/minicluster-data/ts-2/wal/instance:
uuid: "40d6a12e22734200b84df12f2fe0c9c2"
format_stamp: "Formatted at 2026-09-29 17:56:56 on dist-test-slave-53nh"
server_key: "e72e4d151a197509acb9ec2736113736"
server_key_iv: "1eb50673916778f87d77805d8ce6f9a1"
server_key_version: "encryptionkey@0"
I20260929 17:56:56.361035 17771 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 17:56:56.363433 17786 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:56:56.364732 17771 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 17:56:56.364845 17771 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704375975685-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704375975685-392-0/minicluster-data/ts-2/wal
uuid: "40d6a12e22734200b84df12f2fe0c9c2"
format_stamp: "Formatted at 2026-09-29 17:56:56 on dist-test-slave-53nh"
server_key: "e72e4d151a197509acb9ec2736113736"
server_key_iv: "1eb50673916778f87d77805d8ce6f9a1"
server_key_version: "encryptionkey@0"
I20260929 17:56:56.364944 17771 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704375975685-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704375975685-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704375975685-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:56:56.389593 17771 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:56:56.392943 17771 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:56:56.393153 17771 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:56:56.393756 17771 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 17:56:56.394598 17771 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:56:56.394642 17771 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:56:56.394683 17771 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:56:56.394712 17771 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 17:56:56.402912 17771 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45005
I20260929 17:56:56.402944 17899 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45005 every 8 connection(s)
I20260929 17:56:56.403857 17771 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704375975685-392-0/minicluster-data/ts-2/data/info.pb
I20260929 17:56:56.410884 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskup3Soe/build/debug/bin/kudu as pid 17771
I20260929 17:56:56.411036 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704375975685-392-0/minicluster-data/ts-2/wal/instance
I20260929 17:56:56.411345 392 external_mini_cluster.cc:1428] Setting key cd04673f30335f238693c60d1c3b1d1c
Sep 29 17:56:56 dist-test-slave-53nh krb5kdc[17396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704616, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 17:56:56.418497 17480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:56:56.405823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33173 (local address 127.0.98.62:33981)
0929 17:56:56.405997 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0929 17:56:56.406001 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:56:56.406880 (+ 879us) server_negotiation.cc:407] Connection header received
0929 17:56:56.408287 (+ 1407us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:56:56.408291 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:56:56.408366 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:56:56.408445 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:56:56.409799 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:56:56.410337 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:56:56.411908 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:56:56.412155 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:56:56.412249 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:56:56.414947 (+ 2698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:56:56.414950 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:56:56.414952 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:56:56.414979 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:56:56.416460 (+ 1481us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:56:56.417086 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:56:56.417089 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:56:56.417090 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:56:56.417169 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:56:56.417504 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:56:56.417506 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:56:56.417507 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:56:56.417685 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:56:56.417753 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0929 17:56:56.418263 (+ 510us) server_negotiation.cc:299] Negotiation successful
0929 17:56:56.418377 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 17:56:56.419354 17902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:56:56.406236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33981 (local address 127.0.98.3:33173)
0929 17:56:56.406722 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0929 17:56:56.406767 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 17:56:56.408047 (+ 1280us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:56:56.408650 (+ 603us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:56:56.408658 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:56:56.409096 (+ 438us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:56:56.409641 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:56:56.409657 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:56:56.410463 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:56:56.410466 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:56:56.411783 (+ 1317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:56:56.411794 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:56:56.412323 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:56:56.412329 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:56:56.412493 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:56:56.413174 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:56:56.413195 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:56:56.414759 (+ 1564us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:56:56.416619 (+ 1860us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:56:56.416629 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:56:56.416643 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:56:56.416936 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:56:56.417277 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:56:56.417279 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:56:56.417281 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:56:56.417426 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:56:56.417777 (+ 351us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:56:56.417782 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:56:56.418035 (+ 253us) client_negotiation.cc:769] Sending connection context
0929 17:56:56.418196 (+ 161us) client_negotiation.cc:240] Negotiation successful
0929 17:56:56.418509 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":136,"threads_started":1}
I20260929 17:56:56.420459 17900 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33981
I20260929 17:56:56.420722 17900 heartbeater.cc:464] Registering TS with master...
I20260929 17:56:56.421265 17900 heartbeater.cc:511] Master 127.0.98.62:33981 requested a full tablet report, sending...
I20260929 17:56:56.422340 17441 ts_manager.cc:194] Registered new tserver with Master: 40d6a12e22734200b84df12f2fe0c9c2 (127.0.98.3:45005)
I20260929 17:56:56.422931 17441 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:33173
I20260929 17:56:56.425442 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 17:56:56.434823 17480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:56:56.427351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60276 (local address 127.0.98.62:33981)
0929 17:56:56.427501 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0929 17:56:56.427506 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:56:56.427629 (+ 123us) server_negotiation.cc:407] Connection header received
0929 17:56:56.427725 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:56:56.427727 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:56:56.427778 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:56:56.427844 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:56:56.428567 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:56:56.429164 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:56:56.430135 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:56:56.430363 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:56:56.430444 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:56:56.431813 (+ 1369us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:56:56.431816 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:56:56.431818 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:56:56.431841 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:56:56.433246 (+ 1405us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:56:56.433647 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:56:56.433650 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:56:56.433651 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:56:56.433740 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:56:56.433988 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:56:56.433990 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:56:56.433992 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:56:56.434198 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:56:56.434304 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0929 17:56:56.434541 (+ 237us) server_negotiation.cc:299] Negotiation successful
0929 17:56:56.434659 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 17:56:56.437676 17441 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60276:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 17:56:56.440647 17441 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 17:56:56.455606 17911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:56:56.449703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44585 (local address 127.0.0.1:57108)
0929 17:56:56.450294 (+ 591us) negotiation.cc:107] Waiting for socket to connect
0929 17:56:56.450348 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0929 17:56:56.450473 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:56:56.451578 (+ 1105us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:56:56.451583 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:56:56.451604 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:56:56.451735 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:56:56.451740 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:56:56.452993 (+ 1253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:56:56.453016 (+ 23us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:56:56.454295 (+ 1279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:56:56.454304 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:56:56.455054 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:56:56.455058 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:56:56.455191 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:56:56.455233 (+ 42us) client_negotiation.cc:769] Sending connection context
0929 17:56:56.455336 (+ 103us) client_negotiation.cc:240] Negotiation successful
0929 17:56:56.455452 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":415,"mutex_wait_us":35,"thread_start_us":106,"threads_started":1}
I20260929 17:56:56.455610 17913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:56:56.449828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57108 (local address 127.0.98.1:44585)
0929 17:56:56.450996 (+ 1168us) server_negotiation.cc:206] Beginning negotiation
0929 17:56:56.451010 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 17:56:56.451034 (+ 24us) server_negotiation.cc:407] Connection header received
0929 17:56:56.451115 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:56:56.451121 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:56:56.451418 (+ 297us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:56:56.451577 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:56:56.451865 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:56:56.452887 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:56:56.454403 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:56:56.454950 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:56:56.455044 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:56:56.455198 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0929 17:56:56.455375 (+ 177us) server_negotiation.cc:299] Negotiation successful
0929 17:56:56.455467 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1060,"thread_start_us":94,"threads_started":1}
I20260929 17:56:56.455986 17912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:56:56.450498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37843 (local address 127.0.0.1:33980)
0929 17:56:56.451060 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0929 17:56:56.451073 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 17:56:56.451158 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:56:56.451825 (+ 667us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:56:56.451827 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:56:56.451844 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:56:56.451924 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:56:56.451929 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:56:56.453380 (+ 1451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:56:56.453384 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:56:56.454554 (+ 1170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:56:56.454564 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:56:56.455623 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:56:56.455627 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:56:56.455711 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:56:56.455736 (+ 25us) client_negotiation.cc:769] Sending connection context
0929 17:56:56.455786 (+ 50us) client_negotiation.cc:240] Negotiation successful
0929 17:56:56.455851 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":505,"mutex_wait_us":1,"thread_start_us":87,"threads_started":1}
I20260929 17:56:56.456720 17914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:56:56.450156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39878 (local address 127.0.98.3:45005)
0929 17:56:56.451906 (+ 1750us) server_negotiation.cc:206] Beginning negotiation
0929 17:56:56.451918 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 17:56:56.451940 (+ 22us) server_negotiation.cc:407] Connection header received
0929 17:56:56.452013 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:56:56.452018 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:56:56.452263 (+ 245us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:56:56.452403 (+ 140us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:56:56.452621 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:56:56.453871 (+ 1250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:56:56.455235 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:56:56.456169 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:56:56.456264 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:56:56.456397 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0929 17:56:56.456498 (+ 101us) server_negotiation.cc:299] Negotiation successful
0929 17:56:56.456573 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1653,"thread_start_us":146,"threads_started":1}
I20260929 17:56:56.456979 17915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:56:56.449818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45005 (local address 127.0.0.1:39878)
0929 17:56:56.450714 (+ 896us) negotiation.cc:107] Waiting for socket to connect
0929 17:56:56.450724 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 17:56:56.450784 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:56:56.452412 (+ 1628us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:56:56.452415 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:56:56.452433 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:56:56.452515 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:56:56.452520 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:56:56.454013 (+ 1493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:56:56.454017 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:56:56.455105 (+ 1088us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:56:56.455115 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:56:56.456290 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:56:56.456294 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:56:56.456378 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:56:56.456400 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 17:56:56.456465 (+ 65us) client_negotiation.cc:240] Negotiation successful
0929 17:56:56.456860 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":826,"thread_start_us":104,"threads_started":1}
I20260929 17:56:56.456688 17916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:56:56.450668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33980 (local address 127.0.98.2:37843)
0929 17:56:56.451401 (+ 733us) server_negotiation.cc:206] Beginning negotiation
0929 17:56:56.451413 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 17:56:56.451435 (+ 22us) server_negotiation.cc:407] Connection header received
0929 17:56:56.451508 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:56:56.451514 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:56:56.451687 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:56:56.451835 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:56:56.452038 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:56:56.453219 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:56:56.454697 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:56:56.455497 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:56:56.456274 (+ 777us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:56:56.456397 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0929 17:56:56.456446 (+ 49us) server_negotiation.cc:299] Negotiation successful
0929 17:56:56.456536 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":635,"thread_start_us":146,"threads_started":1}
I20260929 17:56:56.458740 17566 tablet_service.cc:1467] Processing CreateTablet for tablet 724e0b6a18034abea90fb63f83b8b3b1 (DEFAULT_TABLE table=test-table [id=44b7002fe12544b8b6fc6fef56bfa8f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 17:56:56.459793 17566 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 724e0b6a18034abea90fb63f83b8b3b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:56:56.459718 17700 tablet_service.cc:1467] Processing CreateTablet for tablet 724e0b6a18034abea90fb63f83b8b3b1 (DEFAULT_TABLE table=test-table [id=44b7002fe12544b8b6fc6fef56bfa8f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 17:56:56.460850 17700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 724e0b6a18034abea90fb63f83b8b3b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:56:56.460891 17834 tablet_service.cc:1467] Processing CreateTablet for tablet 724e0b6a18034abea90fb63f83b8b3b1 (DEFAULT_TABLE table=test-table [id=44b7002fe12544b8b6fc6fef56bfa8f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 17:56:56.461968 17834 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 724e0b6a18034abea90fb63f83b8b3b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:56:56.467028 17917 tablet_bootstrap.cc:492] T 724e0b6a18034abea90fb63f83b8b3b1 P 71aeb1d9d2a540d6807b222a31af7277: Bootstrap starting.
I20260929 17:56:56.468124 17918 tablet_bootstrap.cc:492] T 724e0b6a18034abea90fb63f83b8b3b1 P a9e1d9865cc1424387add2b266e9e2d0: Bootstrap starting.
I20260929 17:56:56.469091 17919 tablet_bootstrap.cc:492] T 724e0b6a18034abea90fb63f83b8b3b1 P 40d6a12e22734200b84df12f2fe0c9c2: Bootstrap starting.
I20260929 17:56:56.469928 17917 tablet_bootstrap.cc:654] T 724e0b6a18034abea90fb63f83b8b3b1 P 71aeb1d9d2a540d6807b222a31af7277: Neither blocks nor log segments found. Creating new log.
I20260929 17:56:56.470405 17918 tablet_bootstrap.cc:654] T 724e0b6a18034abea90fb63f83b8b3b1 P a9e1d9865cc1424387add2b266e9e2d0: Neither blocks nor log segments found. Creating new log.
I20260929 17:56:56.470809 17919 tablet_bootstrap.cc:654] T 724e0b6a18034abea90fb63f83b8b3b1 P 40d6a12e22734200b84df12f2fe0c9c2: Neither blocks nor log segments found. Creating new log.
I20260929 17:56:56.471215 17917 log.cc:824] T 724e0b6a18034abea90fb63f83b8b3b1 P 71aeb1d9d2a540d6807b222a31af7277: Log is configured to *not* fsync() on all Append() calls
I20260929 17:56:56.471540 17918 log.cc:824] T 724e0b6a18034abea90fb63f83b8b3b1 P a9e1d9865cc1424387add2b266e9e2d0: Log is configured to *not* fsync() on all Append() calls
I20260929 17:56:56.471849 17919 log.cc:824] T 724e0b6a18034abea90fb63f83b8b3b1 P 40d6a12e22734200b84df12f2fe0c9c2: Log is configured to *not* fsync() on all Append() calls
I20260929 17:56:56.473703 17918 tablet_bootstrap.cc:492] T 724e0b6a18034abea90fb63f83b8b3b1 P a9e1d9865cc1424387add2b266e9e2d0: No bootstrap required, opened a new log
I20260929 17:56:56.473730 17917 tablet_bootstrap.cc:492] T 724e0b6a18034abea90fb63f83b8b3b1 P 71aeb1d9d2a540d6807b222a31af7277: No bootstrap required, opened a new log
I20260929 17:56:56.473899 17918 ts_tablet_manager.cc:1397] T 724e0b6a18034abea90fb63f83b8b3b1 P a9e1d9865cc1424387add2b266e9e2d0: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260929 17:56:56.473901 17919 tablet_bootstrap.cc:492] T 724e0b6a18034abea90fb63f83b8b3b1 P 40d6a12e22734200b84df12f2fe0c9c2: No bootstrap required, opened a new log
I20260929 17:56:56.473913 17917 ts_tablet_manager.cc:1397] T 724e0b6a18034abea90fb63f83b8b3b1 P 71aeb1d9d2a540d6807b222a31af7277: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260929 17:56:56.474054 17919 ts_tablet_manager.cc:1397] T 724e0b6a18034abea90fb63f83b8b3b1 P 40d6a12e22734200b84df12f2fe0c9c2: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20260929 17:56:56.476771 17918 raft_consensus.cc:348] T 724e0b6a18034abea90fb63f83b8b3b1 P a9e1d9865cc1424387add2b266e9e2d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71aeb1d9d2a540d6807b222a31af7277" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44585 } } peers { permanent_uuid: "40d6a12e22734200b84df12f2fe0c9c2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45005 } } peers { permanent_uuid: "a9e1d9865cc1424387add2b266e9e2d0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37843 } }
I20260929 17:56:56.477054 17918 raft_consensus.cc:374] T 724e0b6a18034abea90fb63f83b8b3b1 P a9e1d9865cc1424387add2b266e9e2d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:56:56.477109 17918 raft_consensus.cc:729] T 724e0b6a18034abea90fb63f83b8b3b1 P a9e1d9865cc1424387add2b266e9e2d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9e1d9865cc1424387add2b266e9e2d0, State: Initialized, Role: FOLLOWER
I20260929 17:56:56.477543 17918 consensus_queue.cc:260] T 724e0b6a18034abea90fb63f83b8b3b1 P a9e1d9865cc1424387add2b266e9e2d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71aeb1d9d2a540d6807b222a31af7277" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44585 } } peers { permanent_uuid: "40d6a12e22734200b84df12f2fe0c9c2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45005 } } peers { permanent_uuid: "a9e1d9865cc1424387add2b266e9e2d0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37843 } }
I20260929 17:56:56.477748 17919 raft_consensus.cc:348] T 724e0b6a18034abea90fb63f83b8b3b1 P 40d6a12e22734200b84df12f2fe0c9c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71aeb1d9d2a540d6807b222a31af7277" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44585 } } peers { permanent_uuid: "40d6a12e22734200b84df12f2fe0c9c2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45005 } } peers { permanent_uuid: "a9e1d9865cc1424387add2b266e9e2d0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37843 } }
I20260929 17:56:56.477749 17917 raft_consensus.cc:348] T 724e0b6a18034abea90fb63f83b8b3b1 P 71aeb1d9d2a540d6807b222a31af7277 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71aeb1d9d2a540d6807b222a31af7277" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44585 } } peers { permanent_uuid: "40d6a12e22734200b84df12f2fe0c9c2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45005 } } peers { permanent_uuid: "a9e1d9865cc1424387add2b266e9e2d0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37843 } }
I20260929 17:56:56.478015 17917 raft_consensus.cc:374] T 724e0b6a18034abea90fb63f83b8b3b1 P 71aeb1d9d2a540d6807b222a31af7277 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:56:56.478015 17919 raft_consensus.cc:374] T 724e0b6a18034abea90fb63f83b8b3b1 P 40d6a12e22734200b84df12f2fe0c9c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:56:56.478075 17917 raft_consensus.cc:729] T 724e0b6a18034abea90fb63f83b8b3b1 P 71aeb1d9d2a540d6807b222a31af7277 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71aeb1d9d2a540d6807b222a31af7277, State: Initialized, Role: FOLLOWER
I20260929 17:56:56.478075 17919 raft_consensus.cc:729] T 724e0b6a18034abea90fb63f83b8b3b1 P 40d6a12e22734200b84df12f2fe0c9c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40d6a12e22734200b84df12f2fe0c9c2, State: Initialized, Role: FOLLOWER
I20260929 17:56:56.478339 17766 heartbeater.cc:503] Master 127.0.98.62:33981 was elected leader, sending a full tablet report...
I20260929 17:56:56.478524 17919 consensus_queue.cc:260] T 724e0b6a18034abea90fb63f83b8b3b1 P 40d6a12e22734200b84df12f2fe0c9c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71aeb1d9d2a540d6807b222a31af7277" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44585 } } peers { permanent_uuid: "40d6a12e22734200b84df12f2fe0c9c2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45005 } } peers { permanent_uuid: "a9e1d9865cc1424387add2b266e9e2d0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37843 } }
I20260929 17:56:56.478516 17917 consensus_queue.cc:260] T 724e0b6a18034abea90fb63f83b8b3b1 P 71aeb1d9d2a540d6807b222a31af7277 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71aeb1d9d2a540d6807b222a31af7277" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44585 } } peers { permanent_uuid: "40d6a12e22734200b84df12f2fe0c9c2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45005 } } peers { permanent_uuid: "a9e1d9865cc1424387add2b266e9e2d0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37843 } }
I20260929 17:56:56.478632 17918 ts_tablet_manager.cc:1428] T 724e0b6a18034abea90fb63f83b8b3b1 P a9e1d9865cc1424387add2b266e9e2d0: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260929 17:56:56.479490 17919 ts_tablet_manager.cc:1428] T 724e0b6a18034abea90fb63f83b8b3b1 P 40d6a12e22734200b84df12f2fe0c9c2: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260929 17:56:56.479490 17917 ts_tablet_manager.cc:1428] T 724e0b6a18034abea90fb63f83b8b3b1 P 71aeb1d9d2a540d6807b222a31af7277: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260929 17:56:56.479765 17632 heartbeater.cc:503] Master 127.0.98.62:33981 was elected leader, sending a full tablet report...
I20260929 17:56:56.479844 17900 heartbeater.cc:503] Master 127.0.98.62:33981 was elected leader, sending a full tablet report...
I20260929 17:56:56.480808 17923 raft_consensus.cc:482] T 724e0b6a18034abea90fb63f83b8b3b1 P a9e1d9865cc1424387add2b266e9e2d0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 17:56:56.480938 17923 raft_consensus.cc:504] T 724e0b6a18034abea90fb63f83b8b3b1 P a9e1d9865cc1424387add2b266e9e2d0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71aeb1d9d2a540d6807b222a31af7277" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44585 } } peers { permanent_uuid: "40d6a12e22734200b84df12f2fe0c9c2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45005 } } peers { permanent_uuid: "a9e1d9865cc1424387add2b266e9e2d0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37843 } }
I20260929 17:56:56.482195 17923 leader_election.cc:288] T 724e0b6a18034abea90fb63f83b8b3b1 P a9e1d9865cc1424387add2b266e9e2d0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 71aeb1d9d2a540d6807b222a31af7277 (127.0.98.1:44585), 40d6a12e22734200b84df12f2fe0c9c2 (127.0.98.3:45005)
I20260929 17:56:56.483410 17925 raft_consensus.cc:482] T 724e0b6a18034abea90fb63f83b8b3b1 P 71aeb1d9d2a540d6807b222a31af7277 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 17:56:56.483566 17925 raft_consensus.cc:504] T 724e0b6a18034abea90fb63f83b8b3b1 P 71aeb1d9d2a540d6807b222a31af7277 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71aeb1d9d2a540d6807b222a31af7277" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44585 } } peers { permanent_uuid: "40d6a12e22734200b84df12f2fe0c9c2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45005 } } peers { permanent_uuid: "a9e1d9865cc1424387add2b266e9e2d0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37843 } }
I20260929 17:56:56.484719 17925 leader_election.cc:288] T 724e0b6a18034abea90fb63f83b8b3b1 P 71aeb1d9d2a540d6807b222a31af7277 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 40d6a12e22734200b84df12f2fe0c9c2 (127.0.98.3:45005), a9e1d9865cc1424387add2b266e9e2d0 (127.0.98.2:37843)
I20260929 17:56:56.487519 17914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:56:56.483264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40445 (local address 127.0.98.3:45005)
0929 17:56:56.483450 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0929 17:56:56.483454 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:56:56.483467 (+ 13us) server_negotiation.cc:407] Connection header received
0929 17:56:56.483507 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:56:56.483510 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:56:56.483550 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:56:56.483616 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:56:56.483961 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:56:56.484673 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:56:56.486381 (+ 1708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:56:56.486837 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:56:56.486936 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:56:56.487000 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0929 17:56:56.487304 (+ 304us) server_negotiation.cc:299] Negotiation successful
0929 17:56:56.487386 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109}
I20260929 17:56:56.487984 17926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:56:56.482585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45005 (local address 127.0.98.2:40445)
0929 17:56:56.482884 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0929 17:56:56.482896 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 17:56:56.482977 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:56:56.483629 (+ 652us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:56:56.483632 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:56:56.483649 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:56:56.483842 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:56:56.483850 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:56:56.484856 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:56:56.484859 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:56:56.486268 (+ 1409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:56:56.486277 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:56:56.487062 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:56:56.487066 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:56:56.487168 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:56:56.487193 (+ 25us) client_negotiation.cc:769] Sending connection context
0929 17:56:56.487748 (+ 555us) client_negotiation.cc:240] Negotiation successful
0929 17:56:56.487832 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"mutex_wait_us":39,"thread_start_us":114,"threads_started":1}
I20260929 17:56:56.488328 17768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:56:56.482383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44585 (local address 127.0.98.2:59317)
0929 17:56:56.482617 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0929 17:56:56.482632 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 17:56:56.482720 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:56:56.483129 (+ 409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:56:56.483132 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:56:56.483151 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:56:56.483274 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:56:56.483281 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:56:56.485598 (+ 2317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:56:56.485602 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:56:56.486800 (+ 1198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:56:56.486810 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:56:56.487935 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:56:56.487939 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:56:56.488047 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:56:56.488070 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 17:56:56.488139 (+ 69us) client_negotiation.cc:240] Negotiation successful
0929 17:56:56.488224 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"mutex_wait_us":32}
I20260929 17:56:56.488615 17854 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "724e0b6a18034abea90fb63f83b8b3b1" candidate_uuid: "a9e1d9865cc1424387add2b266e9e2d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40d6a12e22734200b84df12f2fe0c9c2" is_pre_election: true
I20260929 17:56:56.488952 17854 raft_consensus.cc:2454] T 724e0b6a18034abea90fb63f83b8b3b1 P 40d6a12e22734200b84df12f2fe0c9c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a9e1d9865cc1424387add2b266e9e2d0 in term 0.
I20260929 17:56:56.489205 17913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:56:56.482593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59317 (local address 127.0.98.1:44585)
0929 17:56:56.482935 (+ 342us) server_negotiation.cc:206] Beginning negotiation
0929 17:56:56.482940 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:56:56.482953 (+ 13us) server_negotiation.cc:407] Connection header received
0929 17:56:56.482996 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:56:56.482999 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:56:56.483043 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:56:56.483119 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:56:56.483828 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:56:56.484613 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:56:56.486952 (+ 2339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:56:56.487550 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:56:56.487643 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:56:56.487703 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0929 17:56:56.488967 (+ 1264us) server_negotiation.cc:299] Negotiation successful
0929 17:56:56.489084 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":266}
I20260929 17:56:56.489534 17653 leader_election.cc:302] T 724e0b6a18034abea90fb63f83b8b3b1 P a9e1d9865cc1424387add2b266e9e2d0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 40d6a12e22734200b84df12f2fe0c9c2, a9e1d9865cc1424387add2b266e9e2d0; no voters:
I20260929 17:56:56.489717 17586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "724e0b6a18034abea90fb63f83b8b3b1" candidate_uuid: "a9e1d9865cc1424387add2b266e9e2d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71aeb1d9d2a540d6807b222a31af7277" is_pre_election: true
I20260929 17:56:56.489831 17923 raft_consensus.cc:2781] T 724e0b6a18034abea90fb63f83b8b3b1 P a9e1d9865cc1424387add2b266e9e2d0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 17:56:56.489912 17923 raft_consensus.cc:482] T 724e0b6a18034abea90fb63f83b8b3b1 P a9e1d9865cc1424387add2b266e9e2d0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 17:56:56.489950 17923 raft_consensus.cc:3037] T 724e0b6a18034abea90fb63f83b8b3b1 P a9e1d9865cc1424387add2b266e9e2d0 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:56:56.489924 17586 raft_consensus.cc:2454] T 724e0b6a18034abea90fb63f83b8b3b1 P 71aeb1d9d2a540d6807b222a31af7277 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a9e1d9865cc1424387add2b266e9e2d0 in term 0.
I20260929 17:56:56.490732 17928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:56:56.484920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45005 (local address 127.0.98.1:35345)
0929 17:56:56.485749 (+ 829us) negotiation.cc:107] Waiting for socket to connect
0929 17:56:56.485760 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 17:56:56.485830 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:56:56.487177 (+ 1347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:56:56.487180 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:56:56.487197 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:56:56.487301 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:56:56.487306 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:56:56.488383 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:56:56.488386 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:56:56.489445 (+ 1059us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:56:56.489454 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:56:56.490305 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:56:56.490308 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:56:56.490411 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:56:56.490436 (+ 25us) client_negotiation.cc:769] Sending connection context
0929 17:56:56.490521 (+ 85us) client_negotiation.cc:240] Negotiation successful
0929 17:56:56.490593 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":525,"mutex_wait_us":1}
I20260929 17:56:56.490758 17927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:56:56.485067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35345 (local address 127.0.98.3:45005)
0929 17:56:56.485488 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0929 17:56:56.485497 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 17:56:56.486916 (+ 1419us) server_negotiation.cc:407] Connection header received
0929 17:56:56.486966 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:56:56.486968 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:56:56.487015 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:56:56.487085 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:56:56.487421 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:56:56.488272 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:56:56.489570 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:56:56.490176 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:56:56.490272 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:56:56.490327 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0929 17:56:56.490538 (+ 211us) server_negotiation.cc:299] Negotiation successful
0929 17:56:56.490610 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":142,"threads_started":1}
I20260929 17:56:56.491086 17854 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "724e0b6a18034abea90fb63f83b8b3b1" candidate_uuid: "71aeb1d9d2a540d6807b222a31af7277" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40d6a12e22734200b84df12f2fe0c9c2" is_pre_election: true
I20260929 17:56:56.491207 17854 raft_consensus.cc:2454] T 724e0b6a18034abea90fb63f83b8b3b1 P 40d6a12e22734200b84df12f2fe0c9c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 71aeb1d9d2a540d6807b222a31af7277 in term 0.
I20260929 17:56:56.491268 17923 raft_consensus.cc:504] T 724e0b6a18034abea90fb63f83b8b3b1 P a9e1d9865cc1424387add2b266e9e2d0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71aeb1d9d2a540d6807b222a31af7277" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44585 } } peers { permanent_uuid: "40d6a12e22734200b84df12f2fe0c9c2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45005 } } peers { permanent_uuid: "a9e1d9865cc1424387add2b266e9e2d0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37843 } }
I20260929 17:56:56.491520 17634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:56:56.485186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37843 (local address 127.0.98.1:53677)
0929 17:56:56.485523 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0929 17:56:56.485537 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 17:56:56.485625 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:56:56.485928 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:56:56.485932 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:56:56.485949 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:56:56.486054 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:56:56.486059 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:56:56.487000 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:56:56.487003 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:56:56.488246 (+ 1243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:56:56.488257 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:56:56.491129 (+ 2872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:56:56.491133 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:56:56.491231 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:56:56.491252 (+ 21us) client_negotiation.cc:769] Sending connection context
0929 17:56:56.491317 (+ 65us) client_negotiation.cc:240] Negotiation successful
0929 17:56:56.491392 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":95,"threads_started":1}
I20260929 17:56:56.491528 17916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:56:56.485265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53677 (local address 127.0.98.2:37843)
0929 17:56:56.485357 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0929 17:56:56.485361 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:56:56.485591 (+ 230us) server_negotiation.cc:407] Connection header received
0929 17:56:56.485707 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:56:56.485710 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:56:56.485751 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:56:56.485820 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:56:56.486172 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:56:56.486894 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:56:56.490510 (+ 3616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:56:56.491034 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:56:56.491129 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:56:56.491195 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0929 17:56:56.491339 (+ 144us) server_negotiation.cc:299] Negotiation successful
0929 17:56:56.491403 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260929 17:56:56.491613 17923 leader_election.cc:288] T 724e0b6a18034abea90fb63f83b8b3b1 P a9e1d9865cc1424387add2b266e9e2d0 [CANDIDATE]: Term 1 election: Requested vote from peers 71aeb1d9d2a540d6807b222a31af7277 (127.0.98.1:44585), 40d6a12e22734200b84df12f2fe0c9c2 (127.0.98.3:45005)
I20260929 17:56:56.491833 17519 leader_election.cc:302] T 724e0b6a18034abea90fb63f83b8b3b1 P 71aeb1d9d2a540d6807b222a31af7277 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 40d6a12e22734200b84df12f2fe0c9c2, 71aeb1d9d2a540d6807b222a31af7277; no voters:
I20260929 17:56:56.492105 17720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "724e0b6a18034abea90fb63f83b8b3b1" candidate_uuid: "71aeb1d9d2a540d6807b222a31af7277" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9e1d9865cc1424387add2b266e9e2d0" is_pre_election: true
I20260929 17:56:56.492357 17720 raft_consensus.cc:2379] T 724e0b6a18034abea90fb63f83b8b3b1 P a9e1d9865cc1424387add2b266e9e2d0 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 71aeb1d9d2a540d6807b222a31af7277 in current term 1: Already voted for candidate a9e1d9865cc1424387add2b266e9e2d0 in this term.
I20260929 17:56:56.492344 17854 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "724e0b6a18034abea90fb63f83b8b3b1" candidate_uuid: "a9e1d9865cc1424387add2b266e9e2d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40d6a12e22734200b84df12f2fe0c9c2"
I20260929 17:56:56.492337 17925 raft_consensus.cc:2781] T 724e0b6a18034abea90fb63f83b8b3b1 P 71aeb1d9d2a540d6807b222a31af7277 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 17:56:56.492424 17925 raft_consensus.cc:482] T 724e0b6a18034abea90fb63f83b8b3b1 P 71aeb1d9d2a540d6807b222a31af7277 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 17:56:56.492470 17925 raft_consensus.cc:3037] T 724e0b6a18034abea90fb63f83b8b3b1 P 71aeb1d9d2a540d6807b222a31af7277 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:56:56.492451 17854 raft_consensus.cc:3037] T 724e0b6a18034abea90fb63f83b8b3b1 P 40d6a12e22734200b84df12f2fe0c9c2 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:56:56.492931 17586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "724e0b6a18034abea90fb63f83b8b3b1" candidate_uuid: "a9e1d9865cc1424387add2b266e9e2d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71aeb1d9d2a540d6807b222a31af7277"
I20260929 17:56:56.493681 17854 raft_consensus.cc:2454] T 724e0b6a18034abea90fb63f83b8b3b1 P 40d6a12e22734200b84df12f2fe0c9c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a9e1d9865cc1424387add2b266e9e2d0 in term 1.
I20260929 17:56:56.493702 17925 raft_consensus.cc:504] T 724e0b6a18034abea90fb63f83b8b3b1 P 71aeb1d9d2a540d6807b222a31af7277 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71aeb1d9d2a540d6807b222a31af7277" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44585 } } peers { permanent_uuid: "40d6a12e22734200b84df12f2fe0c9c2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45005 } } peers { permanent_uuid: "a9e1d9865cc1424387add2b266e9e2d0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37843 } }
I20260929 17:56:56.493952 17653 leader_election.cc:302] T 724e0b6a18034abea90fb63f83b8b3b1 P a9e1d9865cc1424387add2b266e9e2d0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 40d6a12e22734200b84df12f2fe0c9c2, a9e1d9865cc1424387add2b266e9e2d0; no voters:
I20260929 17:56:56.494030 17925 leader_election.cc:288] T 724e0b6a18034abea90fb63f83b8b3b1 P 71aeb1d9d2a540d6807b222a31af7277 [CANDIDATE]: Term 1 election: Requested vote from peers 40d6a12e22734200b84df12f2fe0c9c2 (127.0.98.3:45005), a9e1d9865cc1424387add2b266e9e2d0 (127.0.98.2:37843)
I20260929 17:56:56.494127 17923 raft_consensus.cc:2781] T 724e0b6a18034abea90fb63f83b8b3b1 P a9e1d9865cc1424387add2b266e9e2d0 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 17:56:56.494184 17586 raft_consensus.cc:2379] T 724e0b6a18034abea90fb63f83b8b3b1 P 71aeb1d9d2a540d6807b222a31af7277 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate a9e1d9865cc1424387add2b266e9e2d0 in current term 1: Already voted for candidate 71aeb1d9d2a540d6807b222a31af7277 in this term.
I20260929 17:56:56.494362 17923 raft_consensus.cc:686] T 724e0b6a18034abea90fb63f83b8b3b1 P a9e1d9865cc1424387add2b266e9e2d0 [term 1 LEADER]: Becoming Leader. State: Replica: a9e1d9865cc1424387add2b266e9e2d0, State: Running, Role: LEADER
I20260929 17:56:56.494452 17720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "724e0b6a18034abea90fb63f83b8b3b1" candidate_uuid: "71aeb1d9d2a540d6807b222a31af7277" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9e1d9865cc1424387add2b266e9e2d0"
I20260929 17:56:56.494707 17923 consensus_queue.cc:237] T 724e0b6a18034abea90fb63f83b8b3b1 P a9e1d9865cc1424387add2b266e9e2d0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71aeb1d9d2a540d6807b222a31af7277" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44585 } } peers { permanent_uuid: "40d6a12e22734200b84df12f2fe0c9c2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45005 } } peers { permanent_uuid: "a9e1d9865cc1424387add2b266e9e2d0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37843 } }
I20260929 17:56:56.494781 17854 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "724e0b6a18034abea90fb63f83b8b3b1" candidate_uuid: "71aeb1d9d2a540d6807b222a31af7277" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40d6a12e22734200b84df12f2fe0c9c2"
I20260929 17:56:56.494935 17854 raft_consensus.cc:2379] T 724e0b6a18034abea90fb63f83b8b3b1 P 40d6a12e22734200b84df12f2fe0c9c2 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 71aeb1d9d2a540d6807b222a31af7277 in current term 1: Already voted for candidate a9e1d9865cc1424387add2b266e9e2d0 in this term.
I20260929 17:56:56.495337 17519 leader_election.cc:302] T 724e0b6a18034abea90fb63f83b8b3b1 P 71aeb1d9d2a540d6807b222a31af7277 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 71aeb1d9d2a540d6807b222a31af7277; no voters: 40d6a12e22734200b84df12f2fe0c9c2, a9e1d9865cc1424387add2b266e9e2d0
I20260929 17:56:56.495484 17925 raft_consensus.cc:2726] T 724e0b6a18034abea90fb63f83b8b3b1 P 71aeb1d9d2a540d6807b222a31af7277 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260929 17:56:56.497615 17440 catalog_manager.cc:5495] T 724e0b6a18034abea90fb63f83b8b3b1 P a9e1d9865cc1424387add2b266e9e2d0 reported cstate change: term changed from 0 to 1, leader changed from <none> to a9e1d9865cc1424387add2b266e9e2d0 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "a9e1d9865cc1424387add2b266e9e2d0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71aeb1d9d2a540d6807b222a31af7277" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44585 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "40d6a12e22734200b84df12f2fe0c9c2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45005 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a9e1d9865cc1424387add2b266e9e2d0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37843 } health_report { overall_health: HEALTHY } } }
I20260929 17:56:56.520539 17916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:56:56.517315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33990 (local address 127.0.98.2:37843)
0929 17:56:56.517424 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0929 17:56:56.517427 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 17:56:56.517453 (+ 26us) server_negotiation.cc:407] Connection header received
0929 17:56:56.517598 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:56:56.517601 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:56:56.517658 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:56:56.517756 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 17:56:56.517946 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:56:56.518494 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:56:56.519303 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:56:56.519508 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:56:56.519575 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:56:56.519815 (+ 240us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 17:56:56.520231 (+ 416us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 17:56:56.520329 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0929 17:56:56.520354 (+ 25us) server_negotiation.cc:299] Negotiation successful
0929 17:56:56.520402 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260929 17:56:56.526623 17586 raft_consensus.cc:1261] T 724e0b6a18034abea90fb63f83b8b3b1 P 71aeb1d9d2a540d6807b222a31af7277 [term 1 FOLLOWER]: Refusing update from remote peer a9e1d9865cc1424387add2b266e9e2d0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 17:56:56.526621 17854 raft_consensus.cc:1261] T 724e0b6a18034abea90fb63f83b8b3b1 P 40d6a12e22734200b84df12f2fe0c9c2 [term 1 FOLLOWER]: Refusing update from remote peer a9e1d9865cc1424387add2b266e9e2d0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 17:56:56.527281 17929 consensus_queue.cc:1035] T 724e0b6a18034abea90fb63f83b8b3b1 P a9e1d9865cc1424387add2b266e9e2d0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "40d6a12e22734200b84df12f2fe0c9c2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 17:56:56.527464 17923 consensus_queue.cc:1035] T 724e0b6a18034abea90fb63f83b8b3b1 P a9e1d9865cc1424387add2b266e9e2d0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71aeb1d9d2a540d6807b222a31af7277" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 17:56:56.545527 17440 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60276:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 17:56:56.545742 17440 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60276:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 17:56:56.548288 17440 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a40851c12f464ec996f46ef7a79687c6: Sending DeleteTablet for 3 replicas of tablet 724e0b6a18034abea90fb63f83b8b3b1
I20260929 17:56:56.549146 17700 tablet_service.cc:1514] Processing DeleteTablet for tablet 724e0b6a18034abea90fb63f83b8b3b1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 17:56:56 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33980
I20260929 17:56:56.549146 17834 tablet_service.cc:1514] Processing DeleteTablet for tablet 724e0b6a18034abea90fb63f83b8b3b1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 17:56:56 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39878
I20260929 17:56:56.549151 17566 tablet_service.cc:1514] Processing DeleteTablet for tablet 724e0b6a18034abea90fb63f83b8b3b1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 17:56:56 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57108
I20260929 17:56:56.549675 17945 tablet_replica.cc:330] T 724e0b6a18034abea90fb63f83b8b3b1 P 71aeb1d9d2a540d6807b222a31af7277: stopping tablet replica
I20260929 17:56:56.549686 17943 tablet_replica.cc:330] T 724e0b6a18034abea90fb63f83b8b3b1 P a9e1d9865cc1424387add2b266e9e2d0: stopping tablet replica
I20260929 17:56:56.549991 17945 raft_consensus.cc:2229] T 724e0b6a18034abea90fb63f83b8b3b1 P 71aeb1d9d2a540d6807b222a31af7277 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 17:56:56.549993 17943 raft_consensus.cc:2229] T 724e0b6a18034abea90fb63f83b8b3b1 P a9e1d9865cc1424387add2b266e9e2d0 [term 1 LEADER]: Raft consensus shutting down.
I20260929 17:56:56.550316 17945 raft_consensus.cc:2258] T 724e0b6a18034abea90fb63f83b8b3b1 P 71aeb1d9d2a540d6807b222a31af7277 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 17:56:56.550361 17943 raft_consensus.cc:2258] T 724e0b6a18034abea90fb63f83b8b3b1 P a9e1d9865cc1424387add2b266e9e2d0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 17:56:56.550812 17944 tablet_replica.cc:330] T 724e0b6a18034abea90fb63f83b8b3b1 P 40d6a12e22734200b84df12f2fe0c9c2: stopping tablet replica
I20260929 17:56:56.550863 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskup3Soe/build/debug/bin/kudu with pid 17503
I20260929 17:56:56.551067 17944 raft_consensus.cc:2229] T 724e0b6a18034abea90fb63f83b8b3b1 P 40d6a12e22734200b84df12f2fe0c9c2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 17:56:56.551966 17944 raft_consensus.cc:2258] T 724e0b6a18034abea90fb63f83b8b3b1 P 40d6a12e22734200b84df12f2fe0c9c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 17:56:56.553048 17943 ts_tablet_manager.cc:1905] T 724e0b6a18034abea90fb63f83b8b3b1 P a9e1d9865cc1424387add2b266e9e2d0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 17:56:56.553380 17944 ts_tablet_manager.cc:1905] T 724e0b6a18034abea90fb63f83b8b3b1 P 40d6a12e22734200b84df12f2fe0c9c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 17:56:56.555610 17944 ts_tablet_manager.cc:1918] T 724e0b6a18034abea90fb63f83b8b3b1 P 40d6a12e22734200b84df12f2fe0c9c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 17:56:56.555706 17944 log.cc:1196] T 724e0b6a18034abea90fb63f83b8b3b1 P 40d6a12e22734200b84df12f2fe0c9c2: Deleting WAL directory at /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704375975685-392-0/minicluster-data/ts-2/wal/wals/724e0b6a18034abea90fb63f83b8b3b1
I20260929 17:56:56.555778 17943 ts_tablet_manager.cc:1918] T 724e0b6a18034abea90fb63f83b8b3b1 P a9e1d9865cc1424387add2b266e9e2d0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 17:56:56.555871 17943 log.cc:1196] T 724e0b6a18034abea90fb63f83b8b3b1 P a9e1d9865cc1424387add2b266e9e2d0: Deleting WAL directory at /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790704375975685-392-0/minicluster-data/ts-1/wal/wals/724e0b6a18034abea90fb63f83b8b3b1
I20260929 17:56:56.556178 17943 ts_tablet_manager.cc:1939] T 724e0b6a18034abea90fb63f83b8b3b1 P a9e1d9865cc1424387add2b266e9e2d0: Deleting consensus metadata
I20260929 17:56:56.556305 17944 ts_tablet_manager.cc:1939] T 724e0b6a18034abea90fb63f83b8b3b1 P 40d6a12e22734200b84df12f2fe0c9c2: Deleting consensus metadata
I20260929 17:56:56.557355 17425 catalog_manager.cc:4841] TS a9e1d9865cc1424387add2b266e9e2d0 (127.0.98.2:37843): tablet 724e0b6a18034abea90fb63f83b8b3b1 (table test-table [id=44b7002fe12544b8b6fc6fef56bfa8f5]) successfully deleted
I20260929 17:56:56.557569 17425 catalog_manager.cc:4841] TS 40d6a12e22734200b84df12f2fe0c9c2 (127.0.98.3:45005): tablet 724e0b6a18034abea90fb63f83b8b3b1 (table test-table [id=44b7002fe12544b8b6fc6fef56bfa8f5]) successfully deleted
W20260929 17:56:56.560938 17427 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:44585 (error 108)
I20260929 17:56:56.561681 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskup3Soe/build/debug/bin/kudu with pid 17637
I20260929 17:56:56.562453 17911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:56:56.561777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44585 (local address 127.0.0.1:57116)
0929 17:56:56.562278 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0929 17:56:56.562360 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":414}
W20260929 17:56:56.562731 17427 catalog_manager.cc:4584] TS 71aeb1d9d2a540d6807b222a31af7277 (127.0.98.1:44585): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 724e0b6a18034abea90fb63f83b8b3b1: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44585: connect: Connection refused (error 111)
I20260929 17:56:56.567232 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskup3Soe/build/debug/bin/kudu with pid 17771
I20260929 17:56:56.572247 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskup3Soe/build/debug/bin/kudu with pid 17409
2026-09-29T17:56:56Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2156 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-29T17:56:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T17:56:56Z Disabled control of system clock
I20260929 17:56:56.600967 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskup3Soe/build/debug/bin/kudu
/tmp/dist-test-taskup3Soe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790704375975685-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790704375975685-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790704375975685-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790704375975685-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41909
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:46073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41909
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 17:56:56.712066 17950 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:56:56.712368 17950 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:56:56.712407 17950 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:56:56.716408 17950 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 17:56:56.716471 17950 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:56:56.716485 17950 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:56:56.716501 17950 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 17:56:56.716516 17950 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 17:56:56.721647 17950 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790704375975685-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790704375975685-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41909
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41909
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790704375975685-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17950
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790704375975685-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:56:56.722754 17950 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:56:56.724256 17950 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:56:56.733045 17955 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:56:56.733210 17950 server_base.cc:1034] running on GCE node
W20260929 17:56:56.733064 17956 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:56:56.733453 17958 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:56:56.734083 17950 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:56:56.735255 17950 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:56:56.736429 17950 hybrid_clock.cc:648] HybridClock initialized: now 1790704616736403 us; error 36 us; skew 500 ppm
I20260929 17:56:56.738440 17950 webserver.cc:460] Webserver started at http://127.0.98.62:42605/ using document root <none> and password file <none>
I20260929 17:56:56.739013 17950 fs_manager.cc:359] Metadata directory not provided
I20260929 17:56:56.739080 17950 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:56:56.739265 17950 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:56:56.741452 17950 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790704375975685-392-0/minicluster-data/master-0/data/instance:
uuid: "f3befd701bde4f6595d762e410e14a71"
format_stamp: "Formatted at 2026-09-29 17:56:56 on dist-test-slave-53nh"
server_key: "d5271e2c8254d6cdd17d367336cec13c"
server_key_iv: "be6778324f44df2c9fb9f46c0e02caa8"
server_key_version: "encryptionkey@0"
I20260929 17:56:56.742014 17950 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790704375975685-392-0/minicluster-data/master-0/wal/instance:
uuid: "f3befd701bde4f6595d762e410e14a71"
format_stamp: "Formatted at 2026-09-29 17:56:56 on dist-test-slave-53nh"
server_key: "d5271e2c8254d6cdd17d367336cec13c"
server_key_iv: "be6778324f44df2c9fb9f46c0e02caa8"
server_key_version: "encryptionkey@0"
I20260929 17:56:56.745839 17950 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260929 17:56:56.748389 17964 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:56:56.749696 17950 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 17:56:56.749818 17950 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790704375975685-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790704375975685-392-0/minicluster-data/master-0/wal
uuid: "f3befd701bde4f6595d762e410e14a71"
format_stamp: "Formatted at 2026-09-29 17:56:56 on dist-test-slave-53nh"
server_key: "d5271e2c8254d6cdd17d367336cec13c"
server_key_iv: "be6778324f44df2c9fb9f46c0e02caa8"
server_key_version: "encryptionkey@0"
I20260929 17:56:56.749922 17950 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790704375975685-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790704375975685-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790704375975685-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:56:56.778719 17950 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:56:56.779472 17950 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:56:56.779691 17950 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:56:56.786966 17950 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41909
I20260929 17:56:56.786991 18016 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41909 every 8 connection(s)
I20260929 17:56:56.787935 17950 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790704375975685-392-0/minicluster-data/master-0/data/info.pb
I20260929 17:56:56.791096 18017 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:56:56.796916 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskup3Soe/build/debug/bin/kudu as pid 17950
I20260929 17:56:56.796903 18017 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f3befd701bde4f6595d762e410e14a71: Bootstrap starting.
I20260929 17:56:56.797052 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790704375975685-392-0/minicluster-data/master-0/wal/instance
I20260929 17:56:56.797312 392 external_mini_cluster.cc:1428] Setting key ff0d3406a87efce7fb571c591ce4eb16
I20260929 17:56:56.799510 18017 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f3befd701bde4f6595d762e410e14a71: Neither blocks nor log segments found. Creating new log.
I20260929 17:56:56.800246 18017 log.cc:824] T 00000000000000000000000000000000 P f3befd701bde4f6595d762e410e14a71: Log is configured to *not* fsync() on all Append() calls
I20260929 17:56:56.802292 18017 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f3befd701bde4f6595d762e410e14a71: No bootstrap required, opened a new log
I20260929 17:56:56.806579 18017 raft_consensus.cc:348] T 00000000000000000000000000000000 P f3befd701bde4f6595d762e410e14a71 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3befd701bde4f6595d762e410e14a71" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41909 } }
I20260929 17:56:56.806849 18017 raft_consensus.cc:374] T 00000000000000000000000000000000 P f3befd701bde4f6595d762e410e14a71 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:56:56.806906 18017 raft_consensus.cc:729] T 00000000000000000000000000000000 P f3befd701bde4f6595d762e410e14a71 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3befd701bde4f6595d762e410e14a71, State: Initialized, Role: FOLLOWER
I20260929 17:56:56.807255 18020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:56:56.798740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49200 (local address 127.0.98.62:41909)
0929 17:56:56.799381 (+ 641us) server_negotiation.cc:206] Beginning negotiation
0929 17:56:56.799396 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 17:56:56.799433 (+ 37us) server_negotiation.cc:407] Connection header received
0929 17:56:56.800124 (+ 691us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:56:56.800143 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:56:56.800613 (+ 470us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:56:56.801038 (+ 425us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:56:56.801421 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:56:56.802407 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:56:56.803584 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:56:56.804071 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:56:56.804188 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:56:56.805147 (+ 959us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:56:56.805153 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:56:56.805169 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 17:56:56.805624 (+ 455us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:56:56.805691 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 17:56:56.805714 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:56:56.805826 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0929 17:56:56.806007 (+ 181us) server_negotiation.cc:299] Negotiation successful
0929 17:56:56.806341 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272,"thread_start_us":115,"threads_started":1}
I20260929 17:56:56.807377 18017 consensus_queue.cc:260] T 00000000000000000000000000000000 P f3befd701bde4f6595d762e410e14a71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3befd701bde4f6595d762e410e14a71" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41909 } }
I20260929 17:56:56.807545 18017 raft_consensus.cc:388] T 00000000000000000000000000000000 P f3befd701bde4f6595d762e410e14a71 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 17:56:56.807611 18017 raft_consensus.cc:482] T 00000000000000000000000000000000 P f3befd701bde4f6595d762e410e14a71 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 17:56:56.807718 18017 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f3befd701bde4f6595d762e410e14a71 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:56:56.808812 18017 raft_consensus.cc:504] T 00000000000000000000000000000000 P f3befd701bde4f6595d762e410e14a71 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3befd701bde4f6595d762e410e14a71" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41909 } }
I20260929 17:56:56.809180 18017 leader_election.cc:302] T 00000000000000000000000000000000 P f3befd701bde4f6595d762e410e14a71 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f3befd701bde4f6595d762e410e14a71; no voters:
I20260929 17:56:56.809492 18017 leader_election.cc:288] T 00000000000000000000000000000000 P f3befd701bde4f6595d762e410e14a71 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 17:56:56.809612 18022 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f3befd701bde4f6595d762e410e14a71 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 17:56:56.809819 18022 raft_consensus.cc:686] T 00000000000000000000000000000000 P f3befd701bde4f6595d762e410e14a71 [term 1 LEADER]: Becoming Leader. State: Replica: f3befd701bde4f6595d762e410e14a71, State: Running, Role: LEADER
I20260929 17:56:56.810115 18022 consensus_queue.cc:237] T 00000000000000000000000000000000 P f3befd701bde4f6595d762e410e14a71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3befd701bde4f6595d762e410e14a71" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41909 } }
I20260929 17:56:56.810698 18017 sys_catalog.cc:564] T 00000000000000000000000000000000 P f3befd701bde4f6595d762e410e14a71 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 17:56:56.811182 18022 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3befd701bde4f6595d762e410e14a71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f3befd701bde4f6595d762e410e14a71. Latest consensus state: current_term: 1 leader_uuid: "f3befd701bde4f6595d762e410e14a71" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3befd701bde4f6595d762e410e14a71" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41909 } } }
I20260929 17:56:56.811295 18022 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3befd701bde4f6595d762e410e14a71 [sys.catalog]: This master's current role is: LEADER
I20260929 17:56:56.811841 18026 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 17:56:56.811832 18023 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3befd701bde4f6595d762e410e14a71 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f3befd701bde4f6595d762e410e14a71" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3befd701bde4f6595d762e410e14a71" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41909 } } }
I20260929 17:56:56.811931 18023 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3befd701bde4f6595d762e410e14a71 [sys.catalog]: This master's current role is: LEADER
I20260929 17:56:56.814814 18026 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 17:56:56.820438 18026 catalog_manager.cc:1347] Generated new cluster ID: 38e0e6e7ee414bccbcf3e0e6f534cbcc
I20260929 17:56:56.820511 18026 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 17:56:56.828549 18026 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 17:56:56.829840 18026 catalog_manager.cc:1504] Loading token signing keys...
I20260929 17:56:56.839835 18026 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f3befd701bde4f6595d762e410e14a71: Generated new TSK 0
I20260929 17:56:56.840551 18026 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 17:56:56.852475 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskup3Soe/build/debug/bin/kudu
/tmp/dist-test-taskup3Soe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790704375975685-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790704375975685-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790704375975685-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790704375975685-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41909
--builtin_ntp_servers=127.0.98.20:46073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 17:56:56.962587 18041 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:56:56.962857 18041 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:56:56.962895 18041 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:56:56.967068 18041 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:56:56.967152 18041 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:56:56.967219 18041 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 17:56:56.972215 18041 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790704375975685-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790704375975685-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790704375975685-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41909
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18041
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790704375975685-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:56:56.973390 18041 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:56:56.974819 18041 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:56:56.984575 18046 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:56:56.984628 18041 server_base.cc:1034] running on GCE node
W20260929 17:56:56.984637 18047 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:56:56.984570 18049 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:56:56.985275 18041 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:56:56.985950 18041 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:56:56.987121 18041 hybrid_clock.cc:648] HybridClock initialized: now 1790704616987086 us; error 49 us; skew 500 ppm
I20260929 17:56:56.988937 18041 webserver.cc:460] Webserver started at http://127.0.98.1:43303/ using document root <none> and password file <none>
I20260929 17:56:56.989532 18041 fs_manager.cc:359] Metadata directory not provided
I20260929 17:56:56.989594 18041 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:56:56.989787 18041 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:56:56.991866 18041 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790704375975685-392-0/minicluster-data/ts-0/data/instance:
uuid: "cfde4452050f4f229ff4b116a25f3c28"
format_stamp: "Formatted at 2026-09-29 17:56:56 on dist-test-slave-53nh"
server_key: "9c648b94ef3db13a91bd52524694f950"
server_key_iv: "14c7cf9f71f0e621313c7cf0ae388f8d"
server_key_version: "encryptionkey@0"
I20260929 17:56:56.992411 18041 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790704375975685-392-0/minicluster-data/ts-0/wal/instance:
uuid: "cfde4452050f4f229ff4b116a25f3c28"
format_stamp: "Formatted at 2026-09-29 17:56:56 on dist-test-slave-53nh"
server_key: "9c648b94ef3db13a91bd52524694f950"
server_key_iv: "14c7cf9f71f0e621313c7cf0ae388f8d"
server_key_version: "encryptionkey@0"
I20260929 17:56:56.996011 18041 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260929 17:56:56.998446 18055 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:56:56.999729 18041 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 17:56:56.999840 18041 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790704375975685-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790704375975685-392-0/minicluster-data/ts-0/wal
uuid: "cfde4452050f4f229ff4b116a25f3c28"
format_stamp: "Formatted at 2026-09-29 17:56:56 on dist-test-slave-53nh"
server_key: "9c648b94ef3db13a91bd52524694f950"
server_key_iv: "14c7cf9f71f0e621313c7cf0ae388f8d"
server_key_version: "encryptionkey@0"
I20260929 17:56:56.999954 18041 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790704375975685-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790704375975685-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790704375975685-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:56:57.016536 18041 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:56:57.017299 18041 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:56:57.017475 18041 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:56:57.018087 18041 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 17:56:57.019037 18041 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:56:57.019083 18041 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:56:57.019121 18041 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:56:57.019147 18041 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 17:56:57.029103 18041 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38473
I20260929 17:56:57.029127 18168 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38473 every 8 connection(s)
I20260929 17:56:57.030045 18041 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790704375975685-392-0/minicluster-data/ts-0/data/info.pb
I20260929 17:56:57.038749 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskup3Soe/build/debug/bin/kudu as pid 18041
I20260929 17:56:57.038883 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790704375975685-392-0/minicluster-data/ts-0/wal/instance
I20260929 17:56:57.039191 392 external_mini_cluster.cc:1428] Setting key b64ea1bec5179b10bb9778786cbed37a
I20260929 17:56:57.039219 18020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:56:57.031883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44005 (local address 127.0.98.62:41909)
0929 17:56:57.032017 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0929 17:56:57.032021 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:56:57.033186 (+ 1165us) server_negotiation.cc:407] Connection header received
0929 17:56:57.034538 (+ 1352us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:56:57.034541 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:56:57.034588 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:56:57.034663 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:56:57.035527 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:56:57.036069 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:56:57.037262 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:56:57.037494 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:56:57.037576 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:56:57.038686 (+ 1110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:56:57.038690 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:56:57.038692 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 17:56:57.038709 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:56:57.038733 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 17:56:57.038743 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:56:57.038813 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0929 17:56:57.039005 (+ 192us) server_negotiation.cc:299] Negotiation successful
0929 17:56:57.039074 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260929 17:56:57.040102 18171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:56:57.032349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41909 (local address 127.0.98.1:44005)
0929 17:56:57.033040 (+ 691us) negotiation.cc:107] Waiting for socket to connect
0929 17:56:57.033093 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0929 17:56:57.034337 (+ 1244us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:56:57.034831 (+ 494us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:56:57.034838 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:56:57.034922 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:56:57.035409 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:56:57.035420 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:56:57.036183 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:56:57.036185 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:56:57.037152 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:56:57.037161 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:56:57.037617 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:56:57.037620 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:56:57.037774 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:56:57.038400 (+ 626us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 17:56:57.038401 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:56:57.038442 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 17:56:57.038528 (+ 86us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:56:57.038824 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:56:57.038830 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:56:57.038836 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 17:56:57.038989 (+ 153us) client_negotiation.cc:240] Negotiation successful
0929 17:56:57.039218 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":103,"threads_started":1}
I20260929 17:56:57.041182 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskup3Soe/build/debug/bin/kudu
/tmp/dist-test-taskup3Soe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790704375975685-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790704375975685-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790704375975685-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790704375975685-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41909
--builtin_ntp_servers=127.0.98.20:46073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 17:56:57.041209 18169 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41909
I20260929 17:56:57.041435 18169 heartbeater.cc:464] Registering TS with master...
I20260929 17:56:57.041939 18169 heartbeater.cc:511] Master 127.0.98.62:41909 requested a full tablet report, sending...
I20260929 17:56:57.043529 17981 ts_manager.cc:194] Registered new tserver with Master: cfde4452050f4f229ff4b116a25f3c28 (127.0.98.1:38473)
I20260929 17:56:57.045102 17981 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:44005
W20260929 17:56:57.151021 18172 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:56:57.151299 18172 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:56:57.151341 18172 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:56:57.155434 18172 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:56:57.155499 18172 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:56:57.155555 18172 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 17:56:57.160300 18172 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790704375975685-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790704375975685-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790704375975685-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41909
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18172
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790704375975685-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:56:57.161373 18172 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:56:57.162755 18172 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:56:57.172884 18178 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:56:57.172873 18177 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:56:57.172869 18180 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:56:57.172951 18172 server_base.cc:1034] running on GCE node
I20260929 17:56:57.173688 18172 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:56:57.174399 18172 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:56:57.175575 18172 hybrid_clock.cc:648] HybridClock initialized: now 1790704617175559 us; error 38 us; skew 500 ppm
I20260929 17:56:57.177579 18172 webserver.cc:460] Webserver started at http://127.0.98.2:36195/ using document root <none> and password file <none>
I20260929 17:56:57.178154 18172 fs_manager.cc:359] Metadata directory not provided
I20260929 17:56:57.178226 18172 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:56:57.178423 18172 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:56:57.180480 18172 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790704375975685-392-0/minicluster-data/ts-1/data/instance:
uuid: "ad2a57678ede4c93af6c3d1b39bf2577"
format_stamp: "Formatted at 2026-09-29 17:56:57 on dist-test-slave-53nh"
server_key: "f73cf64d457aa6b68f8a17b2492e3666"
server_key_iv: "a821eb82a81209aaf855f0239b4072c1"
server_key_version: "encryptionkey@0"
I20260929 17:56:57.181082 18172 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790704375975685-392-0/minicluster-data/ts-1/wal/instance:
uuid: "ad2a57678ede4c93af6c3d1b39bf2577"
format_stamp: "Formatted at 2026-09-29 17:56:57 on dist-test-slave-53nh"
server_key: "f73cf64d457aa6b68f8a17b2492e3666"
server_key_iv: "a821eb82a81209aaf855f0239b4072c1"
server_key_version: "encryptionkey@0"
I20260929 17:56:57.184693 18172 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260929 17:56:57.187418 18186 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:56:57.188714 18172 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260929 17:56:57.188827 18172 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790704375975685-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790704375975685-392-0/minicluster-data/ts-1/wal
uuid: "ad2a57678ede4c93af6c3d1b39bf2577"
format_stamp: "Formatted at 2026-09-29 17:56:57 on dist-test-slave-53nh"
server_key: "f73cf64d457aa6b68f8a17b2492e3666"
server_key_iv: "a821eb82a81209aaf855f0239b4072c1"
server_key_version: "encryptionkey@0"
I20260929 17:56:57.188944 18172 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790704375975685-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790704375975685-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790704375975685-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:56:57.206875 18172 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:56:57.207686 18172 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:56:57.207866 18172 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:56:57.208498 18172 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 17:56:57.209514 18172 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:56:57.209560 18172 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:56:57.209602 18172 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:56:57.209632 18172 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 17:56:57.218194 18172 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44933
I20260929 17:56:57.218297 18299 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44933 every 8 connection(s)
I20260929 17:56:57.219372 18172 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790704375975685-392-0/minicluster-data/ts-1/data/info.pb
I20260929 17:56:57.228053 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskup3Soe/build/debug/bin/kudu as pid 18172
I20260929 17:56:57.228166 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790704375975685-392-0/minicluster-data/ts-1/wal/instance
I20260929 17:56:57.228325 18020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:56:57.222438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60127 (local address 127.0.98.62:41909)
0929 17:56:57.222586 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0929 17:56:57.222590 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:56:57.222604 (+ 14us) server_negotiation.cc:407] Connection header received
0929 17:56:57.223766 (+ 1162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:56:57.223770 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:56:57.223826 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:56:57.223904 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:56:57.224811 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:56:57.225394 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:56:57.226518 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:56:57.226732 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:56:57.226816 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:56:57.227864 (+ 1048us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:56:57.227867 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:56:57.227869 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 17:56:57.227881 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:56:57.227901 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 17:56:57.227909 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:56:57.227970 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0929 17:56:57.228109 (+ 139us) server_negotiation.cc:299] Negotiation successful
0929 17:56:57.228171 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 17:56:57.228505 392 external_mini_cluster.cc:1428] Setting key dd16dc676f508c9ca5a03d9863041c4c
I20260929 17:56:57.229295 18302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:56:57.221610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41909 (local address 127.0.98.2:60127)
0929 17:56:57.222266 (+ 656us) negotiation.cc:107] Waiting for socket to connect
0929 17:56:57.222315 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 17:56:57.223557 (+ 1242us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:56:57.224069 (+ 512us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:56:57.224075 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:56:57.224162 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:56:57.224676 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:56:57.224690 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:56:57.225519 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:56:57.225523 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:56:57.226406 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:56:57.226415 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:56:57.226876 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:56:57.226880 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:56:57.227033 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:56:57.227600 (+ 567us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 17:56:57.227602 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:56:57.227644 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 17:56:57.227707 (+ 63us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:56:57.227979 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:56:57.227989 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:56:57.227995 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 17:56:57.228101 (+ 106us) client_negotiation.cc:240] Negotiation successful
0929 17:56:57.228384 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":107,"threads_started":1}
I20260929 17:56:57.230384 18300 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41909
I20260929 17:56:57.230459 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskup3Soe/build/debug/bin/kudu
/tmp/dist-test-taskup3Soe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790704375975685-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790704375975685-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790704375975685-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790704375975685-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41909
--builtin_ntp_servers=127.0.98.20:46073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 17:56:57.230623 18300 heartbeater.cc:464] Registering TS with master...
I20260929 17:56:57.231130 18300 heartbeater.cc:511] Master 127.0.98.62:41909 requested a full tablet report, sending...
I20260929 17:56:57.232120 17981 ts_manager.cc:194] Registered new tserver with Master: ad2a57678ede4c93af6c3d1b39bf2577 (127.0.98.2:44933)
I20260929 17:56:57.232690 17981 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:60127
W20260929 17:56:57.341282 18303 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:56:57.341555 18303 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:56:57.341594 18303 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:56:57.345655 18303 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:56:57.345709 18303 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:56:57.345767 18303 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 17:56:57.350533 18303 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790704375975685-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790704375975685-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790704375975685-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41909
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18303
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790704375975685-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:56:57.351558 18303 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:56:57.352959 18303 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:56:57.362062 18309 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:56:57.362200 18303 server_base.cc:1034] running on GCE node
W20260929 17:56:57.362100 18308 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:56:57.362427 18311 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:56:57.362761 18303 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:56:57.363379 18303 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:56:57.364538 18303 hybrid_clock.cc:648] HybridClock initialized: now 1790704617364517 us; error 33 us; skew 500 ppm
I20260929 17:56:57.366407 18303 webserver.cc:460] Webserver started at http://127.0.98.3:36975/ using document root <none> and password file <none>
I20260929 17:56:57.366984 18303 fs_manager.cc:359] Metadata directory not provided
I20260929 17:56:57.367053 18303 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:56:57.367240 18303 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:56:57.369379 18303 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790704375975685-392-0/minicluster-data/ts-2/data/instance:
uuid: "983d4c5bbe4a4492a15541ab79acb998"
format_stamp: "Formatted at 2026-09-29 17:56:57 on dist-test-slave-53nh"
server_key: "0644a9bc455dd10f2ee9512e198c2b48"
server_key_iv: "3885345ad2af8ef74a586b06f0bd267d"
server_key_version: "encryptionkey@0"
I20260929 17:56:57.369920 18303 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790704375975685-392-0/minicluster-data/ts-2/wal/instance:
uuid: "983d4c5bbe4a4492a15541ab79acb998"
format_stamp: "Formatted at 2026-09-29 17:56:57 on dist-test-slave-53nh"
server_key: "0644a9bc455dd10f2ee9512e198c2b48"
server_key_iv: "3885345ad2af8ef74a586b06f0bd267d"
server_key_version: "encryptionkey@0"
I20260929 17:56:57.373437 18303 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260929 17:56:57.375849 18317 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:56:57.377207 18303 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 17:56:57.377315 18303 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790704375975685-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790704375975685-392-0/minicluster-data/ts-2/wal
uuid: "983d4c5bbe4a4492a15541ab79acb998"
format_stamp: "Formatted at 2026-09-29 17:56:57 on dist-test-slave-53nh"
server_key: "0644a9bc455dd10f2ee9512e198c2b48"
server_key_iv: "3885345ad2af8ef74a586b06f0bd267d"
server_key_version: "encryptionkey@0"
I20260929 17:56:57.377404 18303 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790704375975685-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790704375975685-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790704375975685-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:56:57.394451 18303 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:56:57.395110 18303 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:56:57.395278 18303 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:56:57.395825 18303 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 17:56:57.396795 18303 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:56:57.396849 18303 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:56:57.396901 18303 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:56:57.396929 18303 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 17:56:57.404937 18303 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38701
I20260929 17:56:57.404963 18430 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38701 every 8 connection(s)
I20260929 17:56:57.405890 18303 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790704375975685-392-0/minicluster-data/ts-2/data/info.pb
I20260929 17:56:57.406486 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskup3Soe/build/debug/bin/kudu as pid 18303
I20260929 17:56:57.406595 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790704375975685-392-0/minicluster-data/ts-2/wal/instance
I20260929 17:56:57.406878 392 external_mini_cluster.cc:1428] Setting key 2c6e83966f77fb2504c37b0433a60162
I20260929 17:56:57.415629 18020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:56:57.407978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48383 (local address 127.0.98.62:41909)
0929 17:56:57.408124 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0929 17:56:57.408129 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:56:57.409478 (+ 1349us) server_negotiation.cc:407] Connection header received
0929 17:56:57.410866 (+ 1388us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:56:57.410870 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:56:57.410925 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:56:57.410997 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:56:57.411969 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:56:57.412525 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:56:57.413740 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:56:57.413970 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:56:57.414056 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:56:57.415122 (+ 1066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:56:57.415125 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:56:57.415127 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 17:56:57.415138 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:56:57.415159 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 17:56:57.415166 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:56:57.415223 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0929 17:56:57.415411 (+ 188us) server_negotiation.cc:299] Negotiation successful
0929 17:56:57.415478 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 17:56:57.416462 18433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:56:57.408553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41909 (local address 127.0.98.3:48383)
0929 17:56:57.409344 (+ 791us) negotiation.cc:107] Waiting for socket to connect
0929 17:56:57.409390 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 17:56:57.410662 (+ 1272us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:56:57.411236 (+ 574us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:56:57.411244 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:56:57.411343 (+ 99us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:56:57.411838 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:56:57.411852 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:56:57.412646 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:56:57.412649 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:56:57.413627 (+ 978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:56:57.413636 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:56:57.414114 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:56:57.414118 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:56:57.414267 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:56:57.414871 (+ 604us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 17:56:57.414873 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:56:57.414914 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 17:56:57.414978 (+ 64us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:56:57.415231 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:56:57.415240 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:56:57.415246 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 17:56:57.415396 (+ 150us) client_negotiation.cc:240] Negotiation successful
0929 17:56:57.415675 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":108,"threads_started":1}
I20260929 17:56:57.417682 18431 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41909
I20260929 17:56:57.417936 18431 heartbeater.cc:464] Registering TS with master...
I20260929 17:56:57.418444 18431 heartbeater.cc:511] Master 127.0.98.62:41909 requested a full tablet report, sending...
I20260929 17:56:57.419536 17981 ts_manager.cc:194] Registered new tserver with Master: 983d4c5bbe4a4492a15541ab79acb998 (127.0.98.3:38701)
I20260929 17:56:57.420154 17981 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:48383
I20260929 17:56:57.421015 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 17:56:57.426537 18020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:56:57.423065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49204 (local address 127.0.98.62:41909)
0929 17:56:57.423182 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0929 17:56:57.423186 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:56:57.423421 (+ 235us) server_negotiation.cc:407] Connection header received
0929 17:56:57.423522 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:56:57.423524 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:56:57.423563 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:56:57.423633 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:56:57.423924 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:56:57.424439 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:56:57.425250 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:56:57.425465 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:56:57.425549 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:56:57.426203 (+ 654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:56:57.426207 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:56:57.426209 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 17:56:57.426221 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:56:57.426241 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 17:56:57.426249 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:56:57.426307 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0929 17:56:57.426384 (+ 77us) server_negotiation.cc:299] Negotiation successful
0929 17:56:57.426433 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 17:56:57.429088 17981 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49204:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 17:56:57.431969 17981 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 17:56:57.446326 18446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:56:57.440330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38473 (local address 127.0.0.1:34992)
0929 17:56:57.441211 (+ 881us) negotiation.cc:107] Waiting for socket to connect
0929 17:56:57.441224 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 17:56:57.441419 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:56:57.442069 (+ 650us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:56:57.442072 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:56:57.442087 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:56:57.442180 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:56:57.442185 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:56:57.443462 (+ 1277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:56:57.443465 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:56:57.444688 (+ 1223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:56:57.444697 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:56:57.445723 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:56:57.445727 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:56:57.445849 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:56:57.445887 (+ 38us) client_negotiation.cc:769] Sending connection context
0929 17:56:57.446020 (+ 133us) client_negotiation.cc:240] Negotiation successful
0929 17:56:57.446152 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":822,"mutex_wait_us":82,"thread_start_us":115,"threads_started":1}
I20260929 17:56:57.446353 18445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:56:57.440500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34992 (local address 127.0.98.1:38473)
0929 17:56:57.441071 (+ 571us) server_negotiation.cc:206] Beginning negotiation
0929 17:56:57.441084 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 17:56:57.441341 (+ 257us) server_negotiation.cc:407] Connection header received
0929 17:56:57.441568 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:56:57.441575 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:56:57.441772 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:56:57.441900 (+ 128us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:56:57.442288 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:56:57.443367 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:56:57.444812 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:56:57.445594 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:56:57.445709 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:56:57.445935 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0929 17:56:57.446057 (+ 122us) server_negotiation.cc:299] Negotiation successful
0929 17:56:57.446166 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":439,"thread_start_us":110,"threads_started":1}
I20260929 17:56:57.446547 18442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:56:57.440826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44933 (local address 127.0.0.1:55648)
0929 17:56:57.441798 (+ 972us) negotiation.cc:107] Waiting for socket to connect
0929 17:56:57.441823 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0929 17:56:57.441899 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:56:57.442336 (+ 437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:56:57.442339 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:56:57.442355 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:56:57.442440 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:56:57.442446 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:56:57.443365 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:56:57.443368 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:56:57.444954 (+ 1586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:56:57.444965 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:56:57.446162 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:56:57.446166 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:56:57.446256 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:56:57.446280 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 17:56:57.446354 (+ 74us) client_negotiation.cc:240] Negotiation successful
0929 17:56:57.446427 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":914,"mutex_wait_us":124,"thread_start_us":92,"threads_started":1}
I20260929 17:56:57.446715 18443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:56:57.440113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38701 (local address 127.0.0.1:46760)
0929 17:56:57.441123 (+ 1010us) negotiation.cc:107] Waiting for socket to connect
0929 17:56:57.441175 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0929 17:56:57.441312 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:56:57.441852 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:56:57.441857 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:56:57.441879 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:56:57.442003 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:56:57.442009 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:56:57.443385 (+ 1376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:56:57.443388 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:56:57.444954 (+ 1566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:56:57.444965 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:56:57.446036 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:56:57.446040 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:56:57.446132 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:56:57.446151 (+ 19us) client_negotiation.cc:769] Sending connection context
0929 17:56:57.446211 (+ 60us) client_negotiation.cc:240] Negotiation successful
0929 17:56:57.446271 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":793,"mutex_wait_us":168,"thread_start_us":74,"threads_started":1}
I20260929 17:56:57.446969 18447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:56:57.441247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55648 (local address 127.0.98.2:44933)
0929 17:56:57.441667 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0929 17:56:57.441677 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 17:56:57.441984 (+ 307us) server_negotiation.cc:407] Connection header received
0929 17:56:57.442061 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:56:57.442065 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:56:57.442230 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:56:57.442360 (+ 130us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:56:57.442536 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:56:57.443258 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:56:57.445080 (+ 1822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:56:57.445834 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:56:57.445928 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:56:57.446099 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0929 17:56:57.446702 (+ 603us) server_negotiation.cc:299] Negotiation successful
0929 17:56:57.446822 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":333,"thread_start_us":112,"threads_started":1}
I20260929 17:56:57.448112 18444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:56:57.440270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46760 (local address 127.0.98.3:38701)
0929 17:56:57.440827 (+ 557us) server_negotiation.cc:206] Beginning negotiation
0929 17:56:57.440836 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 17:56:57.441454 (+ 618us) server_negotiation.cc:407] Connection header received
0929 17:56:57.441539 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:56:57.441544 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:56:57.441713 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:56:57.441847 (+ 134us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:56:57.442109 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:56:57.443288 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:56:57.445127 (+ 1839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:56:57.445659 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:56:57.445763 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:56:57.445976 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0929 17:56:57.446396 (+ 420us) server_negotiation.cc:299] Negotiation successful
0929 17:56:57.447959 (+ 1563us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":458,"thread_start_us":182,"threads_started":1}
I20260929 17:56:57.449564 18234 tablet_service.cc:1467] Processing CreateTablet for tablet 794fc79580fc4a1e9e9d0e8b1728cb37 (DEFAULT_TABLE table=test-table [id=4affd62698cc434e994bb8ead5a420c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 17:56:57.449574 18103 tablet_service.cc:1467] Processing CreateTablet for tablet 794fc79580fc4a1e9e9d0e8b1728cb37 (DEFAULT_TABLE table=test-table [id=4affd62698cc434e994bb8ead5a420c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 17:56:57.450593 18103 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 794fc79580fc4a1e9e9d0e8b1728cb37. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:56:57.450608 18234 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 794fc79580fc4a1e9e9d0e8b1728cb37. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:56:57.451629 18365 tablet_service.cc:1467] Processing CreateTablet for tablet 794fc79580fc4a1e9e9d0e8b1728cb37 (DEFAULT_TABLE table=test-table [id=4affd62698cc434e994bb8ead5a420c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 17:56:57.452631 18365 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 794fc79580fc4a1e9e9d0e8b1728cb37. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:56:57.456899 18448 tablet_bootstrap.cc:492] T 794fc79580fc4a1e9e9d0e8b1728cb37 P cfde4452050f4f229ff4b116a25f3c28: Bootstrap starting.
I20260929 17:56:57.458353 18449 tablet_bootstrap.cc:492] T 794fc79580fc4a1e9e9d0e8b1728cb37 P ad2a57678ede4c93af6c3d1b39bf2577: Bootstrap starting.
I20260929 17:56:57.459080 18448 tablet_bootstrap.cc:654] T 794fc79580fc4a1e9e9d0e8b1728cb37 P cfde4452050f4f229ff4b116a25f3c28: Neither blocks nor log segments found. Creating new log.
I20260929 17:56:57.459300 18450 tablet_bootstrap.cc:492] T 794fc79580fc4a1e9e9d0e8b1728cb37 P 983d4c5bbe4a4492a15541ab79acb998: Bootstrap starting.
I20260929 17:56:57.460294 18449 tablet_bootstrap.cc:654] T 794fc79580fc4a1e9e9d0e8b1728cb37 P ad2a57678ede4c93af6c3d1b39bf2577: Neither blocks nor log segments found. Creating new log.
I20260929 17:56:57.460302 18448 log.cc:824] T 794fc79580fc4a1e9e9d0e8b1728cb37 P cfde4452050f4f229ff4b116a25f3c28: Log is configured to *not* fsync() on all Append() calls
I20260929 17:56:57.461056 18449 log.cc:824] T 794fc79580fc4a1e9e9d0e8b1728cb37 P ad2a57678ede4c93af6c3d1b39bf2577: Log is configured to *not* fsync() on all Append() calls
I20260929 17:56:57.461428 18450 tablet_bootstrap.cc:654] T 794fc79580fc4a1e9e9d0e8b1728cb37 P 983d4c5bbe4a4492a15541ab79acb998: Neither blocks nor log segments found. Creating new log.
I20260929 17:56:57.462155 18450 log.cc:824] T 794fc79580fc4a1e9e9d0e8b1728cb37 P 983d4c5bbe4a4492a15541ab79acb998: Log is configured to *not* fsync() on all Append() calls
I20260929 17:56:57.462795 18448 tablet_bootstrap.cc:492] T 794fc79580fc4a1e9e9d0e8b1728cb37 P cfde4452050f4f229ff4b116a25f3c28: No bootstrap required, opened a new log
I20260929 17:56:57.462800 18449 tablet_bootstrap.cc:492] T 794fc79580fc4a1e9e9d0e8b1728cb37 P ad2a57678ede4c93af6c3d1b39bf2577: No bootstrap required, opened a new log
I20260929 17:56:57.462972 18449 ts_tablet_manager.cc:1397] T 794fc79580fc4a1e9e9d0e8b1728cb37 P ad2a57678ede4c93af6c3d1b39bf2577: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260929 17:56:57.462981 18448 ts_tablet_manager.cc:1397] T 794fc79580fc4a1e9e9d0e8b1728cb37 P cfde4452050f4f229ff4b116a25f3c28: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260929 17:56:57.463979 18450 tablet_bootstrap.cc:492] T 794fc79580fc4a1e9e9d0e8b1728cb37 P 983d4c5bbe4a4492a15541ab79acb998: No bootstrap required, opened a new log
I20260929 17:56:57.464133 18450 ts_tablet_manager.cc:1397] T 794fc79580fc4a1e9e9d0e8b1728cb37 P 983d4c5bbe4a4492a15541ab79acb998: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260929 17:56:57.466197 18449 raft_consensus.cc:348] T 794fc79580fc4a1e9e9d0e8b1728cb37 P ad2a57678ede4c93af6c3d1b39bf2577 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "983d4c5bbe4a4492a15541ab79acb998" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38701 } } peers { permanent_uuid: "cfde4452050f4f229ff4b116a25f3c28" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38473 } } peers { permanent_uuid: "ad2a57678ede4c93af6c3d1b39bf2577" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44933 } }
I20260929 17:56:57.466607 18449 raft_consensus.cc:374] T 794fc79580fc4a1e9e9d0e8b1728cb37 P ad2a57678ede4c93af6c3d1b39bf2577 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:56:57.466665 18449 raft_consensus.cc:729] T 794fc79580fc4a1e9e9d0e8b1728cb37 P ad2a57678ede4c93af6c3d1b39bf2577 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad2a57678ede4c93af6c3d1b39bf2577, State: Initialized, Role: FOLLOWER
I20260929 17:56:57.467190 18449 consensus_queue.cc:260] T 794fc79580fc4a1e9e9d0e8b1728cb37 P ad2a57678ede4c93af6c3d1b39bf2577 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "983d4c5bbe4a4492a15541ab79acb998" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38701 } } peers { permanent_uuid: "cfde4452050f4f229ff4b116a25f3c28" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38473 } } peers { permanent_uuid: "ad2a57678ede4c93af6c3d1b39bf2577" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44933 } }
I20260929 17:56:57.467450 18448 raft_consensus.cc:348] T 794fc79580fc4a1e9e9d0e8b1728cb37 P cfde4452050f4f229ff4b116a25f3c28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "983d4c5bbe4a4492a15541ab79acb998" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38701 } } peers { permanent_uuid: "cfde4452050f4f229ff4b116a25f3c28" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38473 } } peers { permanent_uuid: "ad2a57678ede4c93af6c3d1b39bf2577" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44933 } }
I20260929 17:56:57.467770 18448 raft_consensus.cc:374] T 794fc79580fc4a1e9e9d0e8b1728cb37 P cfde4452050f4f229ff4b116a25f3c28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:56:57.467860 18448 raft_consensus.cc:729] T 794fc79580fc4a1e9e9d0e8b1728cb37 P cfde4452050f4f229ff4b116a25f3c28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cfde4452050f4f229ff4b116a25f3c28, State: Initialized, Role: FOLLOWER
I20260929 17:56:57.467875 18300 heartbeater.cc:503] Master 127.0.98.62:41909 was elected leader, sending a full tablet report...
I20260929 17:56:57.468216 18449 ts_tablet_manager.cc:1428] T 794fc79580fc4a1e9e9d0e8b1728cb37 P ad2a57678ede4c93af6c3d1b39bf2577: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260929 17:56:57.468356 18448 consensus_queue.cc:260] T 794fc79580fc4a1e9e9d0e8b1728cb37 P cfde4452050f4f229ff4b116a25f3c28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "983d4c5bbe4a4492a15541ab79acb998" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38701 } } peers { permanent_uuid: "cfde4452050f4f229ff4b116a25f3c28" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38473 } } peers { permanent_uuid: "ad2a57678ede4c93af6c3d1b39bf2577" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44933 } }
I20260929 17:56:57.469031 18169 heartbeater.cc:503] Master 127.0.98.62:41909 was elected leader, sending a full tablet report...
I20260929 17:56:57.469321 18450 raft_consensus.cc:348] T 794fc79580fc4a1e9e9d0e8b1728cb37 P 983d4c5bbe4a4492a15541ab79acb998 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "983d4c5bbe4a4492a15541ab79acb998" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38701 } } peers { permanent_uuid: "cfde4452050f4f229ff4b116a25f3c28" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38473 } } peers { permanent_uuid: "ad2a57678ede4c93af6c3d1b39bf2577" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44933 } }
I20260929 17:56:57.469565 18450 raft_consensus.cc:374] T 794fc79580fc4a1e9e9d0e8b1728cb37 P 983d4c5bbe4a4492a15541ab79acb998 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:56:57.469609 18450 raft_consensus.cc:729] T 794fc79580fc4a1e9e9d0e8b1728cb37 P 983d4c5bbe4a4492a15541ab79acb998 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 983d4c5bbe4a4492a15541ab79acb998, State: Initialized, Role: FOLLOWER
I20260929 17:56:57.470031 18450 consensus_queue.cc:260] T 794fc79580fc4a1e9e9d0e8b1728cb37 P 983d4c5bbe4a4492a15541ab79acb998 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "983d4c5bbe4a4492a15541ab79acb998" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38701 } } peers { permanent_uuid: "cfde4452050f4f229ff4b116a25f3c28" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38473 } } peers { permanent_uuid: "ad2a57678ede4c93af6c3d1b39bf2577" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44933 } }
W20260929 17:56:57.470244 18301 tablet.cc:2369] T 794fc79580fc4a1e9e9d0e8b1728cb37 P ad2a57678ede4c93af6c3d1b39bf2577: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 17:56:57.470788 18431 heartbeater.cc:503] Master 127.0.98.62:41909 was elected leader, sending a full tablet report...
I20260929 17:56:57.471184 18450 ts_tablet_manager.cc:1428] T 794fc79580fc4a1e9e9d0e8b1728cb37 P 983d4c5bbe4a4492a15541ab79acb998: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260929 17:56:57.471830 18448 ts_tablet_manager.cc:1428] T 794fc79580fc4a1e9e9d0e8b1728cb37 P cfde4452050f4f229ff4b116a25f3c28: Time spent starting tablet: real 0.009s user 0.006s sys 0.000s
W20260929 17:56:57.531140 18170 tablet.cc:2369] T 794fc79580fc4a1e9e9d0e8b1728cb37 P cfde4452050f4f229ff4b116a25f3c28: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 17:56:57.617830 18454 raft_consensus.cc:482] T 794fc79580fc4a1e9e9d0e8b1728cb37 P ad2a57678ede4c93af6c3d1b39bf2577 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 17:56:57.618036 18454 raft_consensus.cc:504] T 794fc79580fc4a1e9e9d0e8b1728cb37 P ad2a57678ede4c93af6c3d1b39bf2577 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "983d4c5bbe4a4492a15541ab79acb998" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38701 } } peers { permanent_uuid: "cfde4452050f4f229ff4b116a25f3c28" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38473 } } peers { permanent_uuid: "ad2a57678ede4c93af6c3d1b39bf2577" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44933 } }
I20260929 17:56:57.619159 18454 leader_election.cc:288] T 794fc79580fc4a1e9e9d0e8b1728cb37 P ad2a57678ede4c93af6c3d1b39bf2577 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 983d4c5bbe4a4492a15541ab79acb998 (127.0.98.3:38701), cfde4452050f4f229ff4b116a25f3c28 (127.0.98.1:38473)
I20260929 17:56:57.623080 18302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:56:57.619446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38473 (local address 127.0.98.2:55799)
0929 17:56:57.619560 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0929 17:56:57.619575 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 17:56:57.619677 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:56:57.619971 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:56:57.619974 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:56:57.619996 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:56:57.620125 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:56:57.620133 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:56:57.620968 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:56:57.620971 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:56:57.621985 (+ 1014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:56:57.621995 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:56:57.622665 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:56:57.622668 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:56:57.622777 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:56:57.622806 (+ 29us) client_negotiation.cc:769] Sending connection context
0929 17:56:57.622874 (+ 68us) client_negotiation.cc:240] Negotiation successful
0929 17:56:57.622960 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260929 17:56:57.623108 18445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:56:57.619546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55799 (local address 127.0.98.1:38473)
0929 17:56:57.619669 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0929 17:56:57.619673 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:56:57.619688 (+ 15us) server_negotiation.cc:407] Connection header received
0929 17:56:57.619778 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:56:57.619782 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:56:57.619837 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:56:57.619935 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:56:57.620249 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:56:57.620854 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:56:57.622105 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:56:57.622530 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:56:57.622610 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:56:57.622664 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0929 17:56:57.622903 (+ 239us) server_negotiation.cc:299] Negotiation successful
0929 17:56:57.622981 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 17:56:57.623293 18457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:56:57.619446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38701 (local address 127.0.98.2:45183)
0929 17:56:57.619911 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0929 17:56:57.619925 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 17:56:57.620021 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:56:57.620249 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:56:57.620252 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:56:57.620269 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:56:57.620388 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:56:57.620394 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:56:57.621227 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:56:57.621230 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:56:57.622092 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:56:57.622102 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:56:57.622890 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:56:57.622894 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:56:57.623001 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:56:57.623025 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 17:56:57.623099 (+ 74us) client_negotiation.cc:240] Negotiation successful
0929 17:56:57.623164 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":395,"mutex_wait_us":118,"thread_start_us":114,"threads_started":1}
I20260929 17:56:57.623332 18444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:56:57.619546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45183 (local address 127.0.98.3:38701)
0929 17:56:57.619666 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0929 17:56:57.619671 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:56:57.620015 (+ 344us) server_negotiation.cc:407] Connection header received
0929 17:56:57.620110 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:56:57.620113 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:56:57.620160 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:56:57.620230 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:56:57.620486 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:56:57.621099 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:56:57.622228 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:56:57.622682 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:56:57.622868 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:56:57.622950 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0929 17:56:57.623129 (+ 179us) server_negotiation.cc:299] Negotiation successful
0929 17:56:57.623210 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 17:56:57.623963 18123 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "794fc79580fc4a1e9e9d0e8b1728cb37" candidate_uuid: "ad2a57678ede4c93af6c3d1b39bf2577" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cfde4452050f4f229ff4b116a25f3c28" is_pre_election: true
I20260929 17:56:57.623970 18385 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "794fc79580fc4a1e9e9d0e8b1728cb37" candidate_uuid: "ad2a57678ede4c93af6c3d1b39bf2577" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "983d4c5bbe4a4492a15541ab79acb998" is_pre_election: true
I20260929 17:56:57.624243 18385 raft_consensus.cc:2454] T 794fc79580fc4a1e9e9d0e8b1728cb37 P 983d4c5bbe4a4492a15541ab79acb998 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad2a57678ede4c93af6c3d1b39bf2577 in term 0.
I20260929 17:56:57.624255 18123 raft_consensus.cc:2454] T 794fc79580fc4a1e9e9d0e8b1728cb37 P cfde4452050f4f229ff4b116a25f3c28 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad2a57678ede4c93af6c3d1b39bf2577 in term 0.
I20260929 17:56:57.624716 18190 leader_election.cc:302] T 794fc79580fc4a1e9e9d0e8b1728cb37 P ad2a57678ede4c93af6c3d1b39bf2577 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ad2a57678ede4c93af6c3d1b39bf2577, cfde4452050f4f229ff4b116a25f3c28; no voters:
I20260929 17:56:57.625085 18454 raft_consensus.cc:2781] T 794fc79580fc4a1e9e9d0e8b1728cb37 P ad2a57678ede4c93af6c3d1b39bf2577 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 17:56:57.625162 18454 raft_consensus.cc:482] T 794fc79580fc4a1e9e9d0e8b1728cb37 P ad2a57678ede4c93af6c3d1b39bf2577 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 17:56:57.625211 18454 raft_consensus.cc:3037] T 794fc79580fc4a1e9e9d0e8b1728cb37 P ad2a57678ede4c93af6c3d1b39bf2577 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:56:57.626312 18454 raft_consensus.cc:504] T 794fc79580fc4a1e9e9d0e8b1728cb37 P ad2a57678ede4c93af6c3d1b39bf2577 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "983d4c5bbe4a4492a15541ab79acb998" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38701 } } peers { permanent_uuid: "cfde4452050f4f229ff4b116a25f3c28" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38473 } } peers { permanent_uuid: "ad2a57678ede4c93af6c3d1b39bf2577" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44933 } }
I20260929 17:56:57.626660 18454 leader_election.cc:288] T 794fc79580fc4a1e9e9d0e8b1728cb37 P ad2a57678ede4c93af6c3d1b39bf2577 [CANDIDATE]: Term 1 election: Requested vote from peers 983d4c5bbe4a4492a15541ab79acb998 (127.0.98.3:38701), cfde4452050f4f229ff4b116a25f3c28 (127.0.98.1:38473)
I20260929 17:56:57.627031 18123 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "794fc79580fc4a1e9e9d0e8b1728cb37" candidate_uuid: "ad2a57678ede4c93af6c3d1b39bf2577" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cfde4452050f4f229ff4b116a25f3c28"
I20260929 17:56:57.627142 18123 raft_consensus.cc:3037] T 794fc79580fc4a1e9e9d0e8b1728cb37 P cfde4452050f4f229ff4b116a25f3c28 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:56:57.628234 18123 raft_consensus.cc:2454] T 794fc79580fc4a1e9e9d0e8b1728cb37 P cfde4452050f4f229ff4b116a25f3c28 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad2a57678ede4c93af6c3d1b39bf2577 in term 1.
I20260929 17:56:57.628499 18385 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "794fc79580fc4a1e9e9d0e8b1728cb37" candidate_uuid: "ad2a57678ede4c93af6c3d1b39bf2577" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "983d4c5bbe4a4492a15541ab79acb998"
I20260929 17:56:57.628531 18190 leader_election.cc:302] T 794fc79580fc4a1e9e9d0e8b1728cb37 P ad2a57678ede4c93af6c3d1b39bf2577 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ad2a57678ede4c93af6c3d1b39bf2577, cfde4452050f4f229ff4b116a25f3c28; no voters:
I20260929 17:56:57.628612 18385 raft_consensus.cc:3037] T 794fc79580fc4a1e9e9d0e8b1728cb37 P 983d4c5bbe4a4492a15541ab79acb998 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:56:57.629079 18454 raft_consensus.cc:2781] T 794fc79580fc4a1e9e9d0e8b1728cb37 P ad2a57678ede4c93af6c3d1b39bf2577 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 17:56:57.629289 18454 raft_consensus.cc:686] T 794fc79580fc4a1e9e9d0e8b1728cb37 P ad2a57678ede4c93af6c3d1b39bf2577 [term 1 LEADER]: Becoming Leader. State: Replica: ad2a57678ede4c93af6c3d1b39bf2577, State: Running, Role: LEADER
I20260929 17:56:57.629648 18454 consensus_queue.cc:237] T 794fc79580fc4a1e9e9d0e8b1728cb37 P ad2a57678ede4c93af6c3d1b39bf2577 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "983d4c5bbe4a4492a15541ab79acb998" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38701 } } peers { permanent_uuid: "cfde4452050f4f229ff4b116a25f3c28" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38473 } } peers { permanent_uuid: "ad2a57678ede4c93af6c3d1b39bf2577" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44933 } }
I20260929 17:56:57.629709 18385 raft_consensus.cc:2454] T 794fc79580fc4a1e9e9d0e8b1728cb37 P 983d4c5bbe4a4492a15541ab79acb998 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad2a57678ede4c93af6c3d1b39bf2577 in term 1.
I20260929 17:56:57.632854 17980 catalog_manager.cc:5495] T 794fc79580fc4a1e9e9d0e8b1728cb37 P ad2a57678ede4c93af6c3d1b39bf2577 reported cstate change: term changed from 0 to 1, leader changed from <none> to ad2a57678ede4c93af6c3d1b39bf2577 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "ad2a57678ede4c93af6c3d1b39bf2577" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "983d4c5bbe4a4492a15541ab79acb998" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cfde4452050f4f229ff4b116a25f3c28" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38473 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ad2a57678ede4c93af6c3d1b39bf2577" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44933 } health_report { overall_health: HEALTHY } } }
W20260929 17:56:57.657169 18432 tablet.cc:2369] T 794fc79580fc4a1e9e9d0e8b1728cb37 P 983d4c5bbe4a4492a15541ab79acb998: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 17:56:57.686944 18447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:56:57.683304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55650 (local address 127.0.98.2:44933)
0929 17:56:57.683434 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0929 17:56:57.683438 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:56:57.683450 (+ 12us) server_negotiation.cc:407] Connection header received
0929 17:56:57.683525 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:56:57.683528 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:56:57.683588 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:56:57.683673 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 17:56:57.683962 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:56:57.684498 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:56:57.685379 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:56:57.685607 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:56:57.685708 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:56:57.686060 (+ 352us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 17:56:57.686547 (+ 487us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 17:56:57.686646 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0929 17:56:57.686724 (+ 78us) server_negotiation.cc:299] Negotiation successful
0929 17:56:57.686779 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 17:56:57.692910 18385 raft_consensus.cc:1261] T 794fc79580fc4a1e9e9d0e8b1728cb37 P 983d4c5bbe4a4492a15541ab79acb998 [term 1 FOLLOWER]: Refusing update from remote peer ad2a57678ede4c93af6c3d1b39bf2577: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 17:56:57.692910 18123 raft_consensus.cc:1261] T 794fc79580fc4a1e9e9d0e8b1728cb37 P cfde4452050f4f229ff4b116a25f3c28 [term 1 FOLLOWER]: Refusing update from remote peer ad2a57678ede4c93af6c3d1b39bf2577: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 17:56:57.693663 18458 consensus_queue.cc:1035] T 794fc79580fc4a1e9e9d0e8b1728cb37 P ad2a57678ede4c93af6c3d1b39bf2577 [LEADER]: Connected to new peer: Peer: permanent_uuid: "983d4c5bbe4a4492a15541ab79acb998" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 17:56:57.693840 18454 consensus_queue.cc:1035] T 794fc79580fc4a1e9e9d0e8b1728cb37 P ad2a57678ede4c93af6c3d1b39bf2577 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfde4452050f4f229ff4b116a25f3c28" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 17:56:57.709151 17979 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49204:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 17:56:57.709321 17979 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49204:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 17:56:57.711747 17979 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f3befd701bde4f6595d762e410e14a71: Sending DeleteTablet for 3 replicas of tablet 794fc79580fc4a1e9e9d0e8b1728cb37
I20260929 17:56:57.712565 18365 tablet_service.cc:1514] Processing DeleteTablet for tablet 794fc79580fc4a1e9e9d0e8b1728cb37 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 17:56:57 UTC) from {username='slave'} at 127.0.0.1:46760
I20260929 17:56:57.712648 18234 tablet_service.cc:1514] Processing DeleteTablet for tablet 794fc79580fc4a1e9e9d0e8b1728cb37 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 17:56:57 UTC) from {username='slave'} at 127.0.0.1:55648
I20260929 17:56:57.712620 18103 tablet_service.cc:1514] Processing DeleteTablet for tablet 794fc79580fc4a1e9e9d0e8b1728cb37 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 17:56:57 UTC) from {username='slave'} at 127.0.0.1:34992
I20260929 17:56:57.713171 18470 tablet_replica.cc:330] T 794fc79580fc4a1e9e9d0e8b1728cb37 P 983d4c5bbe4a4492a15541ab79acb998: stopping tablet replica
I20260929 17:56:57.713261 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskup3Soe/build/debug/bin/kudu with pid 18041
I20260929 17:56:57.713872 18471 tablet_replica.cc:330] T 794fc79580fc4a1e9e9d0e8b1728cb37 P ad2a57678ede4c93af6c3d1b39bf2577: stopping tablet replica
I20260929 17:56:57.714133 18470 raft_consensus.cc:2229] T 794fc79580fc4a1e9e9d0e8b1728cb37 P 983d4c5bbe4a4492a15541ab79acb998 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 17:56:57.714305 18471 raft_consensus.cc:2229] T 794fc79580fc4a1e9e9d0e8b1728cb37 P ad2a57678ede4c93af6c3d1b39bf2577 [term 1 LEADER]: Raft consensus shutting down.
I20260929 17:56:57.714496 18470 raft_consensus.cc:2258] T 794fc79580fc4a1e9e9d0e8b1728cb37 P 983d4c5bbe4a4492a15541ab79acb998 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 17:56:57.715027 18471 raft_consensus.cc:2258] T 794fc79580fc4a1e9e9d0e8b1728cb37 P ad2a57678ede4c93af6c3d1b39bf2577 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 17:56:57.716435 18471 ts_tablet_manager.cc:1905] T 794fc79580fc4a1e9e9d0e8b1728cb37 P ad2a57678ede4c93af6c3d1b39bf2577: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 17:56:57.717177 18470 ts_tablet_manager.cc:1905] T 794fc79580fc4a1e9e9d0e8b1728cb37 P 983d4c5bbe4a4492a15541ab79acb998: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 17:56:57.719278 18471 ts_tablet_manager.cc:1918] T 794fc79580fc4a1e9e9d0e8b1728cb37 P ad2a57678ede4c93af6c3d1b39bf2577: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 17:56:57.719461 18471 log.cc:1196] T 794fc79580fc4a1e9e9d0e8b1728cb37 P ad2a57678ede4c93af6c3d1b39bf2577: Deleting WAL directory at /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790704375975685-392-0/minicluster-data/ts-1/wal/wals/794fc79580fc4a1e9e9d0e8b1728cb37
I20260929 17:56:57.719816 18471 ts_tablet_manager.cc:1939] T 794fc79580fc4a1e9e9d0e8b1728cb37 P ad2a57678ede4c93af6c3d1b39bf2577: Deleting consensus metadata
I20260929 17:56:57.719815 18470 ts_tablet_manager.cc:1918] T 794fc79580fc4a1e9e9d0e8b1728cb37 P 983d4c5bbe4a4492a15541ab79acb998: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 17:56:57.719900 18470 log.cc:1196] T 794fc79580fc4a1e9e9d0e8b1728cb37 P 983d4c5bbe4a4492a15541ab79acb998: Deleting WAL directory at /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790704375975685-392-0/minicluster-data/ts-2/wal/wals/794fc79580fc4a1e9e9d0e8b1728cb37
I20260929 17:56:57.720202 18470 ts_tablet_manager.cc:1939] T 794fc79580fc4a1e9e9d0e8b1728cb37 P 983d4c5bbe4a4492a15541ab79acb998: Deleting consensus metadata
W20260929 17:56:57.721333 17968 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:38473 (error 108)
I20260929 17:56:57.721388 17967 catalog_manager.cc:4841] TS 983d4c5bbe4a4492a15541ab79acb998 (127.0.98.3:38701): tablet 794fc79580fc4a1e9e9d0e8b1728cb37 (table test-table [id=4affd62698cc434e994bb8ead5a420c6]) successfully deleted
I20260929 17:56:57.721397 17965 catalog_manager.cc:4841] TS ad2a57678ede4c93af6c3d1b39bf2577 (127.0.98.2:44933): tablet 794fc79580fc4a1e9e9d0e8b1728cb37 (table test-table [id=4affd62698cc434e994bb8ead5a420c6]) successfully deleted
I20260929 17:56:57.722141 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskup3Soe/build/debug/bin/kudu with pid 18172
I20260929 17:56:57.722400 18446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:56:57.722121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38473 (local address 127.0.0.1:34996)
0929 17:56:57.722249 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0929 17:56:57.722313 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38473: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260929 17:56:57.723647 17968 catalog_manager.cc:4584] TS cfde4452050f4f229ff4b116a25f3c28 (127.0.98.1:38473): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 794fc79580fc4a1e9e9d0e8b1728cb37: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38473: connect: Connection refused (error 111)
I20260929 17:56:57.727581 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskup3Soe/build/debug/bin/kudu with pid 18303
I20260929 17:56:57.733287 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskup3Soe/build/debug/bin/kudu with pid 17950
2026-09-29T17:56:57Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (1159 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704375975685-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 17:56:57 dist-test-slave-53nh krb5kdc[18476](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 17:56:57 dist-test-slave-53nh krb5kdc[18476](info): set up 2 sockets
Sep 29 17:56:57 dist-test-slave-53nh krb5kdc[18476](info): commencing operation
krb5kdc: starting...
W20260929 17:56:58.688889 392 mini_kdc.cc:121] Time spent starting KDC: real 0.935s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704375975685-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704375975685-392-0/krb5kdc/test-user.keytab.
Sep 29 17:56:58 dist-test-slave-53nh krb5kdc[18476](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704618, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T17:56:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T17:56:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704375975685-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704375975685-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 17:56:58.808918 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskup3Soe/build/debug/bin/kudu
/tmp/dist-test-taskup3Soe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704375975685-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704375975685-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704375975685-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704375975685-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38745
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704375975685-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38745
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704375975685-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704375975685-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704375975685-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:56:58.920536 18489 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:56:58.920822 18489 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:56:58.920876 18489 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:56:58.924937 18489 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 17:56:58.925040 18489 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:56:58.925068 18489 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:56:58.925099 18489 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 17:56:58.925128 18489 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 17:56:58.930832 18489 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704375975685-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704375975685-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38745
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704375975685-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38745
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704375975685-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18489
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704375975685-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:56:58.931988 18489 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:56:58.933110 18489 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:56:58.941623 18497 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:56:58.941633 18495 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:56:58.941670 18489 server_base.cc:1034] running on GCE node
W20260929 17:56:58.941633 18494 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:56:58.942456 18489 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:56:58.943615 18489 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:56:58.944802 18489 hybrid_clock.cc:648] HybridClock initialized: now 1790704618944783 us; error 49 us; skew 500 ppm
Sep 29 17:56:58 dist-test-slave-53nh krb5kdc[18476](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704618, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:56:58.955996 18489 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 17:56:58.956977 18489 webserver.cc:460] Webserver started at http://127.0.98.62:44759/ using document root <none> and password file <none>
I20260929 17:56:58.957573 18489 fs_manager.cc:359] Metadata directory not provided
I20260929 17:56:58.957638 18489 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:56:58.957803 18489 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:56:58.959860 18489 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704375975685-392-0/minicluster-data/master-0/data/instance:
uuid: "092f892a1adf4a4793bfe28230dd9611"
format_stamp: "Formatted at 2026-09-29 17:56:58 on dist-test-slave-53nh"
server_key: "9654e783010d96a6e2ce68aa14e9d65a"
server_key_iv: "e8224b3b02ab874153dadecdb49f7c54"
server_key_version: "encryptionkey@0"
I20260929 17:56:58.960399 18489 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704375975685-392-0/minicluster-data/master-0/wal/instance:
uuid: "092f892a1adf4a4793bfe28230dd9611"
format_stamp: "Formatted at 2026-09-29 17:56:58 on dist-test-slave-53nh"
server_key: "9654e783010d96a6e2ce68aa14e9d65a"
server_key_iv: "e8224b3b02ab874153dadecdb49f7c54"
server_key_version: "encryptionkey@0"
I20260929 17:56:58.964313 18489 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 17:56:58.967093 18504 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:56:58.968632 18489 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260929 17:56:58.968778 18489 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704375975685-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704375975685-392-0/minicluster-data/master-0/wal
uuid: "092f892a1adf4a4793bfe28230dd9611"
format_stamp: "Formatted at 2026-09-29 17:56:58 on dist-test-slave-53nh"
server_key: "9654e783010d96a6e2ce68aa14e9d65a"
server_key_iv: "e8224b3b02ab874153dadecdb49f7c54"
server_key_version: "encryptionkey@0"
I20260929 17:56:58.968922 18489 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704375975685-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704375975685-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704375975685-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:56:58.988469 18489 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:56:58.991961 18489 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:56:58.992154 18489 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:56:58.999084 18489 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38745
I20260929 17:56:58.999101 18556 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38745 every 8 connection(s)
I20260929 17:56:59.000108 18489 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704375975685-392-0/minicluster-data/master-0/data/info.pb
I20260929 17:56:59.003079 18557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:56:59.005612 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskup3Soe/build/debug/bin/kudu as pid 18489
I20260929 17:56:59.005750 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704375975685-392-0/minicluster-data/master-0/wal/instance
I20260929 17:56:59.006044 392 external_mini_cluster.cc:1428] Setting key bc7ecda92b27bc8cc8e442803ec3fc70
I20260929 17:56:59.010094 18557 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 092f892a1adf4a4793bfe28230dd9611: Bootstrap starting.
I20260929 17:56:59.012902 18557 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 092f892a1adf4a4793bfe28230dd9611: Neither blocks nor log segments found. Creating new log.
I20260929 17:56:59.013861 18557 log.cc:824] T 00000000000000000000000000000000 P 092f892a1adf4a4793bfe28230dd9611: Log is configured to *not* fsync() on all Append() calls
Sep 29 17:56:59 dist-test-slave-53nh krb5kdc[18476](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704618, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 17:56:59.016224 18557 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 092f892a1adf4a4793bfe28230dd9611: No bootstrap required, opened a new log
I20260929 17:56:59.020478 18557 raft_consensus.cc:348] T 00000000000000000000000000000000 P 092f892a1adf4a4793bfe28230dd9611 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "092f892a1adf4a4793bfe28230dd9611" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38745 } }
I20260929 17:56:59.020781 18557 raft_consensus.cc:374] T 00000000000000000000000000000000 P 092f892a1adf4a4793bfe28230dd9611 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:56:59.020838 18557 raft_consensus.cc:729] T 00000000000000000000000000000000 P 092f892a1adf4a4793bfe28230dd9611 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 092f892a1adf4a4793bfe28230dd9611, State: Initialized, Role: FOLLOWER
I20260929 17:56:59.021386 18557 consensus_queue.cc:260] T 00000000000000000000000000000000 P 092f892a1adf4a4793bfe28230dd9611 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "092f892a1adf4a4793bfe28230dd9611" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38745 } }
I20260929 17:56:59.021580 18557 raft_consensus.cc:388] T 00000000000000000000000000000000 P 092f892a1adf4a4793bfe28230dd9611 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 17:56:59.021658 18557 raft_consensus.cc:482] T 00000000000000000000000000000000 P 092f892a1adf4a4793bfe28230dd9611 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 17:56:59.021763 18557 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 092f892a1adf4a4793bfe28230dd9611 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:56:59.023000 18557 raft_consensus.cc:504] T 00000000000000000000000000000000 P 092f892a1adf4a4793bfe28230dd9611 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "092f892a1adf4a4793bfe28230dd9611" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38745 } }
I20260929 17:56:59.023402 18557 leader_election.cc:302] T 00000000000000000000000000000000 P 092f892a1adf4a4793bfe28230dd9611 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 092f892a1adf4a4793bfe28230dd9611; no voters:
I20260929 17:56:59.023734 18560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:56:59.007558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33274 (local address 127.0.98.62:38745)
0929 17:56:59.008057 (+ 499us) server_negotiation.cc:206] Beginning negotiation
0929 17:56:59.008078 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0929 17:56:59.008126 (+ 48us) server_negotiation.cc:407] Connection header received
0929 17:56:59.008991 (+ 865us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:56:59.009054 (+ 63us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:56:59.009523 (+ 469us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:56:59.009935 (+ 412us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:56:59.010669 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:56:59.012178 (+ 1509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:56:59.013201 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:56:59.013919 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:56:59.014089 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:56:59.016722 (+ 2633us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:56:59.016730 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:56:59.016749 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:56:59.016783 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:56:59.019367 (+ 2584us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:56:59.019939 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:56:59.019946 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:56:59.019977 (+ 31us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:56:59.020130 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:56:59.020489 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:56:59.020493 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:56:59.020495 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:56:59.021081 (+ 586us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:56:59.021322 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0929 17:56:59.022327 (+ 1005us) server_negotiation.cc:299] Negotiation successful
0929 17:56:59.022652 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":127,"threads_started":1}
I20260929 17:56:59.023939 18562 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 092f892a1adf4a4793bfe28230dd9611 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 17:56:59.023748 18557 leader_election.cc:288] T 00000000000000000000000000000000 P 092f892a1adf4a4793bfe28230dd9611 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 17:56:59.024149 18562 raft_consensus.cc:686] T 00000000000000000000000000000000 P 092f892a1adf4a4793bfe28230dd9611 [term 1 LEADER]: Becoming Leader. State: Replica: 092f892a1adf4a4793bfe28230dd9611, State: Running, Role: LEADER
I20260929 17:56:59.024472 18562 consensus_queue.cc:237] T 00000000000000000000000000000000 P 092f892a1adf4a4793bfe28230dd9611 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "092f892a1adf4a4793bfe28230dd9611" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38745 } }
I20260929 17:56:59.025378 18557 sys_catalog.cc:564] T 00000000000000000000000000000000 P 092f892a1adf4a4793bfe28230dd9611 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 17:56:59.025727 18562 sys_catalog.cc:455] T 00000000000000000000000000000000 P 092f892a1adf4a4793bfe28230dd9611 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "092f892a1adf4a4793bfe28230dd9611" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "092f892a1adf4a4793bfe28230dd9611" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38745 } } }
I20260929 17:56:59.025836 18562 sys_catalog.cc:458] T 00000000000000000000000000000000 P 092f892a1adf4a4793bfe28230dd9611 [sys.catalog]: This master's current role is: LEADER
I20260929 17:56:59.027004 18564 sys_catalog.cc:455] T 00000000000000000000000000000000 P 092f892a1adf4a4793bfe28230dd9611 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 092f892a1adf4a4793bfe28230dd9611. Latest consensus state: current_term: 1 leader_uuid: "092f892a1adf4a4793bfe28230dd9611" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "092f892a1adf4a4793bfe28230dd9611" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38745 } } }
I20260929 17:56:59.027235 18564 sys_catalog.cc:458] T 00000000000000000000000000000000 P 092f892a1adf4a4793bfe28230dd9611 [sys.catalog]: This master's current role is: LEADER
I20260929 17:56:59.027998 18566 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 17:56:59.031563 18566 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 17:56:59.036612 18566 catalog_manager.cc:1347] Generated new cluster ID: 75416eb05a1847408feb3ab1e2536de5
I20260929 17:56:59.036681 18566 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 17:56:59.066411 18566 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 17:56:59.067585 18566 catalog_manager.cc:1504] Loading token signing keys...
I20260929 17:56:59.077384 18566 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 092f892a1adf4a4793bfe28230dd9611: Generated new TSK 0
I20260929 17:56:59.078154 18566 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704375975685-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704375975685-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260929 17:56:59.116083 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskup3Soe/build/debug/bin/kudu
/tmp/dist-test-taskup3Soe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704375975685-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704375975685-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704375975685-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704375975685-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38745
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704375975685-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704375975685-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704375975685-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704375975685-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:56:59.226979 18583 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:56:59.227241 18583 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:56:59.227285 18583 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:56:59.231565 18583 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:56:59.231634 18583 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:56:59.231720 18583 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 17:56:59.237142 18583 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704375975685-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704375975685-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704375975685-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704375975685-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38745
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18583
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704375975685-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:56:59.238196 18583 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:56:59.239269 18583 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:56:59.249099 18591 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:56:59.249099 18588 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:56:59.249099 18589 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:56:59.249315 18583 server_base.cc:1034] running on GCE node
I20260929 17:56:59.249910 18583 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:56:59.250633 18583 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:56:59.251803 18583 hybrid_clock.cc:648] HybridClock initialized: now 1790704619251780 us; error 46 us; skew 500 ppm
Sep 29 17:56:59 dist-test-slave-53nh krb5kdc[18476](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704619, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:56:59.254711 18583 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260929 17:56:59.255632 18583 webserver.cc:460] Webserver started at http://127.0.98.1:46789/ using document root <none> and password file <none>
I20260929 17:56:59.256192 18583 fs_manager.cc:359] Metadata directory not provided
I20260929 17:56:59.256266 18583 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:56:59.256474 18583 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:56:59.258563 18583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704375975685-392-0/minicluster-data/ts-0/data/instance:
uuid: "60ea964e26264206b5841a1e1deb3d9d"
format_stamp: "Formatted at 2026-09-29 17:56:59 on dist-test-slave-53nh"
server_key: "aa1825afff877d93cd47a076569ee294"
server_key_iv: "7a26a5276f76c557661fece89cb2b72b"
server_key_version: "encryptionkey@0"
I20260929 17:56:59.259096 18583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704375975685-392-0/minicluster-data/ts-0/wal/instance:
uuid: "60ea964e26264206b5841a1e1deb3d9d"
format_stamp: "Formatted at 2026-09-29 17:56:59 on dist-test-slave-53nh"
server_key: "aa1825afff877d93cd47a076569ee294"
server_key_iv: "7a26a5276f76c557661fece89cb2b72b"
server_key_version: "encryptionkey@0"
I20260929 17:56:59.262840 18583 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260929 17:56:59.265455 18598 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:56:59.266649 18583 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 17:56:59.266743 18583 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704375975685-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704375975685-392-0/minicluster-data/ts-0/wal
uuid: "60ea964e26264206b5841a1e1deb3d9d"
format_stamp: "Formatted at 2026-09-29 17:56:59 on dist-test-slave-53nh"
server_key: "aa1825afff877d93cd47a076569ee294"
server_key_iv: "7a26a5276f76c557661fece89cb2b72b"
server_key_version: "encryptionkey@0"
I20260929 17:56:59.266829 18583 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704375975685-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704375975685-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704375975685-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:56:59.286906 18583 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:56:59.290131 18583 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:56:59.290292 18583 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:56:59.290854 18583 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 17:56:59.291663 18583 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:56:59.291715 18583 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:56:59.291754 18583 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:56:59.291774 18583 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 17:56:59.299979 18583 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33347
I20260929 17:56:59.300042 18711 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33347 every 8 connection(s)
I20260929 17:56:59.300858 18583 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704375975685-392-0/minicluster-data/ts-0/data/info.pb
I20260929 17:56:59.302407 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskup3Soe/build/debug/bin/kudu as pid 18583
I20260929 17:56:59.302541 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704375975685-392-0/minicluster-data/ts-0/wal/instance
I20260929 17:56:59.302831 392 external_mini_cluster.cc:1428] Setting key 80320f85d5ad57b9e76d8a5c7cb4c8be
Sep 29 17:56:59 dist-test-slave-53nh krb5kdc[18476](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704619, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260929 17:56:59.314711 18560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:56:59.302674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48497 (local address 127.0.98.62:38745)
0929 17:56:59.302828 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0929 17:56:59.302832 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:56:59.303736 (+ 904us) server_negotiation.cc:407] Connection header received
0929 17:56:59.305249 (+ 1513us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:56:59.305252 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:56:59.305294 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:56:59.305359 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:56:59.306691 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:56:59.307489 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:56:59.308633 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:56:59.308985 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:56:59.309121 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:56:59.311375 (+ 2254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:56:59.311379 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:56:59.311382 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:56:59.311408 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:56:59.312740 (+ 1332us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:56:59.313245 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:56:59.313249 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:56:59.313250 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:56:59.313323 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:56:59.313641 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:56:59.313644 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:56:59.313646 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:56:59.313823 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:56:59.313904 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0929 17:56:59.314409 (+ 505us) server_negotiation.cc:299] Negotiation successful
0929 17:56:59.314547 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 17:56:59.315512 18714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:56:59.303072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38745 (local address 127.0.98.1:48497)
0929 17:56:59.303607 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0929 17:56:59.303650 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 17:56:59.305027 (+ 1377us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:56:59.305545 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:56:59.305554 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:56:59.306036 (+ 482us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:56:59.306544 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:56:59.306558 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:56:59.307601 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:56:59.307604 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:56:59.308517 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:56:59.308526 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:56:59.309139 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:56:59.309141 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:56:59.309276 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:56:59.309868 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:56:59.309884 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:56:59.311207 (+ 1323us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:56:59.312845 (+ 1638us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:56:59.312851 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:56:59.312865 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:56:59.313158 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:56:59.313402 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:56:59.313404 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:56:59.313406 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:56:59.313556 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:56:59.313915 (+ 359us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:56:59.313921 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:56:59.314173 (+ 252us) client_negotiation.cc:769] Sending connection context
0929 17:56:59.314357 (+ 184us) client_negotiation.cc:240] Negotiation successful
0929 17:56:59.314676 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":140,"threads_started":1}
I20260929 17:56:59.316733 18712 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38745
I20260929 17:56:59.317028 18712 heartbeater.cc:464] Registering TS with master...
I20260929 17:56:59.317608 18712 heartbeater.cc:511] Master 127.0.98.62:38745 requested a full tablet report, sending...
I20260929 17:56:59.319285 18521 ts_manager.cc:194] Registered new tserver with Master: 60ea964e26264206b5841a1e1deb3d9d (127.0.98.1:33347)
I20260929 17:56:59.320420 18521 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:48497
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704375975685-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704375975685-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260929 17:56:59.332252 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskup3Soe/build/debug/bin/kudu
/tmp/dist-test-taskup3Soe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704375975685-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704375975685-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704375975685-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704375975685-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38745
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704375975685-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704375975685-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704375975685-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704375975685-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:56:59.441955 18717 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:56:59.442227 18717 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:56:59.442272 18717 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:56:59.446360 18717 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:56:59.446434 18717 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:56:59.446504 18717 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 17:56:59.452095 18717 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704375975685-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704375975685-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704375975685-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704375975685-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38745
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18717
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704375975685-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:56:59.453275 18717 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:56:59.454285 18717 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:56:59.464098 18725 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:56:59.464098 18723 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:56:59.464098 18722 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:56:59.464406 18717 server_base.cc:1034] running on GCE node
I20260929 17:56:59.464831 18717 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:56:59.465502 18717 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:56:59.466684 18717 hybrid_clock.cc:648] HybridClock initialized: now 1790704619466654 us; error 42 us; skew 500 ppm
Sep 29 17:56:59 dist-test-slave-53nh krb5kdc[18476](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704619, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:56:59.469417 18717 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260929 17:56:59.470288 18717 webserver.cc:460] Webserver started at http://127.0.98.2:38811/ using document root <none> and password file <none>
I20260929 17:56:59.470901 18717 fs_manager.cc:359] Metadata directory not provided
I20260929 17:56:59.470963 18717 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:56:59.471133 18717 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:56:59.473309 18717 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704375975685-392-0/minicluster-data/ts-1/data/instance:
uuid: "1af4cbe8231e4dcfbe9d2d9effe611b6"
format_stamp: "Formatted at 2026-09-29 17:56:59 on dist-test-slave-53nh"
server_key: "a3b82e519f27cad5e0fee279f8fd3bbb"
server_key_iv: "5fdbaa8ceaf10f88c52f663de6472235"
server_key_version: "encryptionkey@0"
I20260929 17:56:59.473872 18717 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704375975685-392-0/minicluster-data/ts-1/wal/instance:
uuid: "1af4cbe8231e4dcfbe9d2d9effe611b6"
format_stamp: "Formatted at 2026-09-29 17:56:59 on dist-test-slave-53nh"
server_key: "a3b82e519f27cad5e0fee279f8fd3bbb"
server_key_iv: "5fdbaa8ceaf10f88c52f663de6472235"
server_key_version: "encryptionkey@0"
I20260929 17:56:59.477964 18717 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260929 17:56:59.480330 18732 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:56:59.481675 18717 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 17:56:59.481791 18717 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704375975685-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704375975685-392-0/minicluster-data/ts-1/wal
uuid: "1af4cbe8231e4dcfbe9d2d9effe611b6"
format_stamp: "Formatted at 2026-09-29 17:56:59 on dist-test-slave-53nh"
server_key: "a3b82e519f27cad5e0fee279f8fd3bbb"
server_key_iv: "5fdbaa8ceaf10f88c52f663de6472235"
server_key_version: "encryptionkey@0"
I20260929 17:56:59.481897 18717 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704375975685-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704375975685-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704375975685-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:56:59.492897 18717 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:56:59.496289 18717 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:56:59.496469 18717 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:56:59.497138 18717 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 17:56:59.497983 18717 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:56:59.498034 18717 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:56:59.498075 18717 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:56:59.498092 18717 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 17:56:59.506054 18717 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35193
I20260929 17:56:59.506085 18845 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35193 every 8 connection(s)
I20260929 17:56:59.506932 18717 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704375975685-392-0/minicluster-data/ts-1/data/info.pb
I20260929 17:56:59.508383 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskup3Soe/build/debug/bin/kudu as pid 18717
I20260929 17:56:59.508484 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704375975685-392-0/minicluster-data/ts-1/wal/instance
I20260929 17:56:59.508798 392 external_mini_cluster.cc:1428] Setting key 8992047bb50de0ffcad4c853d2d71191
Sep 29 17:56:59 dist-test-slave-53nh krb5kdc[18476](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704619, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260929 17:56:59.521807 18560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:56:59.508710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46099 (local address 127.0.98.62:38745)
0929 17:56:59.508855 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0929 17:56:59.508859 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:56:59.509824 (+ 965us) server_negotiation.cc:407] Connection header received
0929 17:56:59.511346 (+ 1522us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:56:59.511348 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:56:59.511392 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:56:59.511459 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:56:59.512842 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:56:59.513523 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:56:59.515187 (+ 1664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:56:59.515414 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:56:59.515496 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:56:59.518361 (+ 2865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:56:59.518365 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:56:59.518368 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:56:59.518397 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:56:59.519685 (+ 1288us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:56:59.520146 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:56:59.520149 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:56:59.520150 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:56:59.520221 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:56:59.520521 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:56:59.520524 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:56:59.520526 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:56:59.520739 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:56:59.520822 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0929 17:56:59.521523 (+ 701us) server_negotiation.cc:299] Negotiation successful
0929 17:56:59.521648 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 17:56:59.522727 18848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:56:59.509183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38745 (local address 127.0.98.2:46099)
0929 17:56:59.509686 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0929 17:56:59.509734 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0929 17:56:59.511112 (+ 1378us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:56:59.511697 (+ 585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:56:59.511706 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:56:59.512185 (+ 479us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:56:59.512677 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:56:59.512691 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:56:59.513656 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:56:59.513659 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:56:59.515050 (+ 1391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:56:59.515061 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:56:59.515576 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:56:59.515580 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:56:59.515804 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:56:59.516420 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:56:59.516441 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:56:59.518181 (+ 1740us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:56:59.519789 (+ 1608us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:56:59.519794 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:56:59.519807 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:56:59.520047 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:56:59.520302 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:56:59.520305 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:56:59.520307 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:56:59.520436 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:56:59.520831 (+ 395us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:56:59.520838 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:56:59.521242 (+ 404us) client_negotiation.cc:769] Sending connection context
0929 17:56:59.521450 (+ 208us) client_negotiation.cc:240] Negotiation successful
0929 17:56:59.521735 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":143,"threads_started":1}
I20260929 17:56:59.524084 18846 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38745
I20260929 17:56:59.524318 18846 heartbeater.cc:464] Registering TS with master...
I20260929 17:56:59.524852 18846 heartbeater.cc:511] Master 127.0.98.62:38745 requested a full tablet report, sending...
I20260929 17:56:59.525940 18521 ts_manager.cc:194] Registered new tserver with Master: 1af4cbe8231e4dcfbe9d2d9effe611b6 (127.0.98.2:35193)
I20260929 17:56:59.526741 18521 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:46099
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704375975685-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704375975685-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260929 17:56:59.535699 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskup3Soe/build/debug/bin/kudu
/tmp/dist-test-taskup3Soe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704375975685-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704375975685-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704375975685-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704375975685-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38745
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704375975685-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704375975685-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704375975685-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704375975685-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:56:59.646095 18851 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:56:59.646379 18851 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:56:59.646435 18851 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:56:59.650568 18851 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:56:59.650657 18851 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:56:59.650730 18851 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 17:56:59.656246 18851 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704375975685-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704375975685-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704375975685-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704375975685-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38745
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18851
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704375975685-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:56:59.657358 18851 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:56:59.658368 18851 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 17:56:59.667667 18851 server_base.cc:1034] running on GCE node
W20260929 17:56:59.667654 18856 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:56:59.667675 18857 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:56:59.667614 18859 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:56:59.668339 18851 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:56:59.668934 18851 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:56:59.670082 18851 hybrid_clock.cc:648] HybridClock initialized: now 1790704619670062 us; error 30 us; skew 500 ppm
Sep 29 17:56:59 dist-test-slave-53nh krb5kdc[18476](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704619, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:56:59.672766 18851 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260929 17:56:59.673694 18851 webserver.cc:460] Webserver started at http://127.0.98.3:40119/ using document root <none> and password file <none>
I20260929 17:56:59.674238 18851 fs_manager.cc:359] Metadata directory not provided
I20260929 17:56:59.674300 18851 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:56:59.674502 18851 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:56:59.676515 18851 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704375975685-392-0/minicluster-data/ts-2/data/instance:
uuid: "dd63bce3e264450b87284d5ed110b4ee"
format_stamp: "Formatted at 2026-09-29 17:56:59 on dist-test-slave-53nh"
server_key: "3cbd047611b3fc6af56be4369fa3c0cd"
server_key_iv: "23e93be09efb676684b2dc4d528d2da7"
server_key_version: "encryptionkey@0"
I20260929 17:56:59.677078 18851 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704375975685-392-0/minicluster-data/ts-2/wal/instance:
uuid: "dd63bce3e264450b87284d5ed110b4ee"
format_stamp: "Formatted at 2026-09-29 17:56:59 on dist-test-slave-53nh"
server_key: "3cbd047611b3fc6af56be4369fa3c0cd"
server_key_iv: "23e93be09efb676684b2dc4d528d2da7"
server_key_version: "encryptionkey@0"
I20260929 17:56:59.680689 18851 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260929 17:56:59.683073 18866 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:56:59.684315 18851 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 17:56:59.684419 18851 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704375975685-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704375975685-392-0/minicluster-data/ts-2/wal
uuid: "dd63bce3e264450b87284d5ed110b4ee"
format_stamp: "Formatted at 2026-09-29 17:56:59 on dist-test-slave-53nh"
server_key: "3cbd047611b3fc6af56be4369fa3c0cd"
server_key_iv: "23e93be09efb676684b2dc4d528d2da7"
server_key_version: "encryptionkey@0"
I20260929 17:56:59.684566 18851 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704375975685-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704375975685-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704375975685-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:56:59.699878 18851 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:56:59.702919 18851 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:56:59.703089 18851 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:56:59.703720 18851 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 17:56:59.704546 18851 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:56:59.704591 18851 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:56:59.704632 18851 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:56:59.704660 18851 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 17:56:59.712646 18851 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36563
I20260929 17:56:59.712694 18979 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36563 every 8 connection(s)
I20260929 17:56:59.713544 18851 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704375975685-392-0/minicluster-data/ts-2/data/info.pb
I20260929 17:56:59.721885 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskup3Soe/build/debug/bin/kudu as pid 18851
I20260929 17:56:59.722015 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704375975685-392-0/minicluster-data/ts-2/wal/instance
I20260929 17:56:59.722275 392 external_mini_cluster.cc:1428] Setting key 16972e5c3b99d640df41ce1cb589eae7
Sep 29 17:56:59 dist-test-slave-53nh krb5kdc[18476](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704619, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 17:56:59.727346 18560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:56:59.715275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57253 (local address 127.0.98.62:38745)
0929 17:56:59.715409 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0929 17:56:59.715412 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 17:56:59.716287 (+ 875us) server_negotiation.cc:407] Connection header received
0929 17:56:59.717667 (+ 1380us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:56:59.717670 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:56:59.717728 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:56:59.717804 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:56:59.718991 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:56:59.719584 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:56:59.720723 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:56:59.720956 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:56:59.721095 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:56:59.723754 (+ 2659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:56:59.723757 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:56:59.723759 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:56:59.723784 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:56:59.725213 (+ 1429us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:56:59.725715 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:56:59.725718 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:56:59.725720 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:56:59.725851 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:56:59.726245 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:56:59.726248 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:56:59.726249 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:56:59.726473 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:56:59.726559 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0929 17:56:59.727108 (+ 549us) server_negotiation.cc:299] Negotiation successful
0929 17:56:59.727221 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 17:56:59.728279 18982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:56:59.715654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38745 (local address 127.0.98.3:57253)
0929 17:56:59.716111 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0929 17:56:59.716163 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0929 17:56:59.717432 (+ 1269us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:56:59.717994 (+ 562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:56:59.718002 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:56:59.718355 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:56:59.718850 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:56:59.718864 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:56:59.719704 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:56:59.719707 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:56:59.720609 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:56:59.720619 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:56:59.721134 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:56:59.721138 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:56:59.721305 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:56:59.721883 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:56:59.721903 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:56:59.723602 (+ 1699us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:56:59.725303 (+ 1701us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:56:59.725309 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:56:59.725323 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:56:59.725605 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:56:59.725965 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:56:59.725968 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:56:59.725970 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:56:59.726160 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:56:59.726567 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:56:59.726573 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:56:59.726863 (+ 290us) client_negotiation.cc:769] Sending connection context
0929 17:56:59.727072 (+ 209us) client_negotiation.cc:240] Negotiation successful
0929 17:56:59.727373 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":114,"threads_started":1}
I20260929 17:56:59.729589 18980 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38745
I20260929 17:56:59.729897 18980 heartbeater.cc:464] Registering TS with master...
I20260929 17:56:59.730450 18980 heartbeater.cc:511] Master 127.0.98.62:38745 requested a full tablet report, sending...
I20260929 17:56:59.731617 18521 ts_manager.cc:194] Registered new tserver with Master: dd63bce3e264450b87284d5ed110b4ee (127.0.98.3:36563)
I20260929 17:56:59.732182 18521 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:57253
I20260929 17:56:59.736080 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 17:56:59.745189 18560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:56:59.738057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33286 (local address 127.0.98.62:38745)
0929 17:56:59.738198 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0929 17:56:59.738203 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:56:59.738335 (+ 132us) server_negotiation.cc:407] Connection header received
0929 17:56:59.738501 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:56:59.738504 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:56:59.738544 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:56:59.738605 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:56:59.739297 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:56:59.739814 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:56:59.740621 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:56:59.740839 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:56:59.740919 (+ 80us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:56:59.742370 (+ 1451us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:56:59.742374 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:56:59.742377 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:56:59.742403 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:56:59.743492 (+ 1089us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:56:59.743944 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:56:59.743947 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:56:59.743949 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:56:59.744021 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:56:59.744322 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:56:59.744326 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:56:59.744328 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:56:59.744536 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:56:59.744664 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0929 17:56:59.744909 (+ 245us) server_negotiation.cc:299] Negotiation successful
0929 17:56:59.745042 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 17:56:59.748050 18521 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33286:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 17:56:59.751186 18521 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 17:56:59.765775 18995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:56:59.759815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33347 (local address 127.0.0.1:58440)
0929 17:56:59.760573 (+ 758us) negotiation.cc:107] Waiting for socket to connect
0929 17:56:59.760584 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 17:56:59.760666 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:56:59.761648 (+ 982us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:56:59.761651 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:56:59.761680 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:56:59.761807 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:56:59.761813 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:56:59.763345 (+ 1532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:56:59.763348 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:56:59.764151 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:56:59.764161 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:56:59.765214 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:56:59.765218 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:56:59.765334 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:56:59.765371 (+ 37us) client_negotiation.cc:769] Sending connection context
0929 17:56:59.765476 (+ 105us) client_negotiation.cc:240] Negotiation successful
0929 17:56:59.765603 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":705,"mutex_wait_us":19,"spinlock_wait_cycles":6912,"thread_start_us":97,"threads_started":1}
I20260929 17:56:59.765915 18996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:56:59.760576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56636 (local address 127.0.98.2:35193)
0929 17:56:59.760831 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0929 17:56:59.760842 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 17:56:59.760964 (+ 122us) server_negotiation.cc:407] Connection header received
0929 17:56:59.761112 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:56:59.761118 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:56:59.761315 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:56:59.761459 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:56:59.762058 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:56:59.762793 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:56:59.764525 (+ 1732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:56:59.765353 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:56:59.765462 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:56:59.765607 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0929 17:56:59.765656 (+ 49us) server_negotiation.cc:299] Negotiation successful
0929 17:56:59.765753 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":74,"threads_started":1}
I20260929 17:56:59.765777 18991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:56:59.760364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35193 (local address 127.0.0.1:56636)
0929 17:56:59.760874 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0929 17:56:59.760885 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 17:56:59.760968 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:56:59.761648 (+ 680us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:56:59.761651 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:56:59.761682 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:56:59.761958 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:56:59.761963 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:56:59.762918 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:56:59.762921 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:56:59.764379 (+ 1458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:56:59.764388 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:56:59.765456 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:56:59.765460 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:56:59.765539 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:56:59.765560 (+ 21us) client_negotiation.cc:769] Sending connection context
0929 17:56:59.765623 (+ 63us) client_negotiation.cc:240] Negotiation successful
0929 17:56:59.765681 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":455,"mutex_wait_us":230,"spinlock_wait_cycles":2432,"thread_start_us":90,"threads_started":1}
I20260929 17:56:59.765986 18992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:56:59.759689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36563 (local address 127.0.0.1:33876)
0929 17:56:59.760489 (+ 800us) negotiation.cc:107] Waiting for socket to connect
0929 17:56:59.760540 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0929 17:56:59.760666 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:56:59.761460 (+ 794us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:56:59.761466 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:56:59.761490 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:56:59.761633 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:56:59.761640 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:56:59.763022 (+ 1382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:56:59.763025 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:56:59.764378 (+ 1353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:56:59.764388 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:56:59.765613 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:56:59.765617 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:56:59.765704 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:56:59.765726 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 17:56:59.765771 (+ 45us) client_negotiation.cc:240] Negotiation successful
0929 17:56:59.765837 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":623,"mutex_wait_us":70,"thread_start_us":101,"threads_started":1}
I20260929 17:56:59.766433 18993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:56:59.759819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33876 (local address 127.0.98.3:36563)
0929 17:56:59.760929 (+ 1110us) server_negotiation.cc:206] Beginning negotiation
0929 17:56:59.760941 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 17:56:59.760962 (+ 21us) server_negotiation.cc:407] Connection header received
0929 17:56:59.761076 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:56:59.761081 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:56:59.761309 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:56:59.761458 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:56:59.761757 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:56:59.762904 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:56:59.764527 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:56:59.765490 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:56:59.765916 (+ 426us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:56:59.766142 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0929 17:56:59.766194 (+ 52us) server_negotiation.cc:299] Negotiation successful
0929 17:56:59.766289 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1017,"thread_start_us":92,"threads_started":1}
I20260929 17:56:59.766716 18994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:56:59.760194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58440 (local address 127.0.98.1:33347)
0929 17:56:59.761250 (+ 1056us) server_negotiation.cc:206] Beginning negotiation
0929 17:56:59.761261 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 17:56:59.761281 (+ 20us) server_negotiation.cc:407] Connection header received
0929 17:56:59.761348 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:56:59.761352 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:56:59.761507 (+ 155us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:56:59.762092 (+ 585us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:56:59.762253 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:56:59.763226 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:56:59.764288 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:56:59.765109 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:56:59.765204 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:56:59.765362 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0929 17:56:59.766401 (+ 1039us) server_negotiation.cc:299] Negotiation successful
0929 17:56:59.766572 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":957,"thread_start_us":115,"threads_started":1}
I20260929 17:56:59.769235 18780 tablet_service.cc:1467] Processing CreateTablet for tablet cbde07c7e0cf4af2b17b16ca92b1847f (DEFAULT_TABLE table=test-table [id=9f6d05fc01e546ab9cfcc0be4e609313]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 17:56:59.769321 18914 tablet_service.cc:1467] Processing CreateTablet for tablet cbde07c7e0cf4af2b17b16ca92b1847f (DEFAULT_TABLE table=test-table [id=9f6d05fc01e546ab9cfcc0be4e609313]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 17:56:59.769321 18646 tablet_service.cc:1467] Processing CreateTablet for tablet cbde07c7e0cf4af2b17b16ca92b1847f (DEFAULT_TABLE table=test-table [id=9f6d05fc01e546ab9cfcc0be4e609313]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 17:56:59.770332 18780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cbde07c7e0cf4af2b17b16ca92b1847f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:56:59.770339 18646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cbde07c7e0cf4af2b17b16ca92b1847f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:56:59.770339 18914 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cbde07c7e0cf4af2b17b16ca92b1847f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:56:59.776484 18997 tablet_bootstrap.cc:492] T cbde07c7e0cf4af2b17b16ca92b1847f P 1af4cbe8231e4dcfbe9d2d9effe611b6: Bootstrap starting.
I20260929 17:56:59.777971 18998 tablet_bootstrap.cc:492] T cbde07c7e0cf4af2b17b16ca92b1847f P 60ea964e26264206b5841a1e1deb3d9d: Bootstrap starting.
I20260929 17:56:59.778542 18999 tablet_bootstrap.cc:492] T cbde07c7e0cf4af2b17b16ca92b1847f P dd63bce3e264450b87284d5ed110b4ee: Bootstrap starting.
I20260929 17:56:59.779263 18997 tablet_bootstrap.cc:654] T cbde07c7e0cf4af2b17b16ca92b1847f P 1af4cbe8231e4dcfbe9d2d9effe611b6: Neither blocks nor log segments found. Creating new log.
I20260929 17:56:59.780110 18998 tablet_bootstrap.cc:654] T cbde07c7e0cf4af2b17b16ca92b1847f P 60ea964e26264206b5841a1e1deb3d9d: Neither blocks nor log segments found. Creating new log.
I20260929 17:56:59.780227 18999 tablet_bootstrap.cc:654] T cbde07c7e0cf4af2b17b16ca92b1847f P dd63bce3e264450b87284d5ed110b4ee: Neither blocks nor log segments found. Creating new log.
I20260929 17:56:59.780582 18997 log.cc:824] T cbde07c7e0cf4af2b17b16ca92b1847f P 1af4cbe8231e4dcfbe9d2d9effe611b6: Log is configured to *not* fsync() on all Append() calls
I20260929 17:56:59.781315 18999 log.cc:824] T cbde07c7e0cf4af2b17b16ca92b1847f P dd63bce3e264450b87284d5ed110b4ee: Log is configured to *not* fsync() on all Append() calls
I20260929 17:56:59.781324 18998 log.cc:824] T cbde07c7e0cf4af2b17b16ca92b1847f P 60ea964e26264206b5841a1e1deb3d9d: Log is configured to *not* fsync() on all Append() calls
I20260929 17:56:59.782804 18997 tablet_bootstrap.cc:492] T cbde07c7e0cf4af2b17b16ca92b1847f P 1af4cbe8231e4dcfbe9d2d9effe611b6: No bootstrap required, opened a new log
I20260929 17:56:59.783028 18997 ts_tablet_manager.cc:1397] T cbde07c7e0cf4af2b17b16ca92b1847f P 1af4cbe8231e4dcfbe9d2d9effe611b6: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.003s
I20260929 17:56:59.783159 18998 tablet_bootstrap.cc:492] T cbde07c7e0cf4af2b17b16ca92b1847f P 60ea964e26264206b5841a1e1deb3d9d: No bootstrap required, opened a new log
I20260929 17:56:59.783265 18999 tablet_bootstrap.cc:492] T cbde07c7e0cf4af2b17b16ca92b1847f P dd63bce3e264450b87284d5ed110b4ee: No bootstrap required, opened a new log
I20260929 17:56:59.783340 18998 ts_tablet_manager.cc:1397] T cbde07c7e0cf4af2b17b16ca92b1847f P 60ea964e26264206b5841a1e1deb3d9d: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260929 17:56:59.783442 18999 ts_tablet_manager.cc:1397] T cbde07c7e0cf4af2b17b16ca92b1847f P dd63bce3e264450b87284d5ed110b4ee: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260929 17:56:59.786518 18999 raft_consensus.cc:348] T cbde07c7e0cf4af2b17b16ca92b1847f P dd63bce3e264450b87284d5ed110b4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd63bce3e264450b87284d5ed110b4ee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36563 } } peers { permanent_uuid: "60ea964e26264206b5841a1e1deb3d9d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33347 } } peers { permanent_uuid: "1af4cbe8231e4dcfbe9d2d9effe611b6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35193 } }
I20260929 17:56:59.786825 18999 raft_consensus.cc:374] T cbde07c7e0cf4af2b17b16ca92b1847f P dd63bce3e264450b87284d5ed110b4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:56:59.786885 18999 raft_consensus.cc:729] T cbde07c7e0cf4af2b17b16ca92b1847f P dd63bce3e264450b87284d5ed110b4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd63bce3e264450b87284d5ed110b4ee, State: Initialized, Role: FOLLOWER
I20260929 17:56:59.786952 18998 raft_consensus.cc:348] T cbde07c7e0cf4af2b17b16ca92b1847f P 60ea964e26264206b5841a1e1deb3d9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd63bce3e264450b87284d5ed110b4ee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36563 } } peers { permanent_uuid: "60ea964e26264206b5841a1e1deb3d9d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33347 } } peers { permanent_uuid: "1af4cbe8231e4dcfbe9d2d9effe611b6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35193 } }
I20260929 17:56:59.786952 18997 raft_consensus.cc:348] T cbde07c7e0cf4af2b17b16ca92b1847f P 1af4cbe8231e4dcfbe9d2d9effe611b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd63bce3e264450b87284d5ed110b4ee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36563 } } peers { permanent_uuid: "60ea964e26264206b5841a1e1deb3d9d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33347 } } peers { permanent_uuid: "1af4cbe8231e4dcfbe9d2d9effe611b6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35193 } }
I20260929 17:56:59.787201 18997 raft_consensus.cc:374] T cbde07c7e0cf4af2b17b16ca92b1847f P 1af4cbe8231e4dcfbe9d2d9effe611b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:56:59.787201 18998 raft_consensus.cc:374] T cbde07c7e0cf4af2b17b16ca92b1847f P 60ea964e26264206b5841a1e1deb3d9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:56:59.787253 18998 raft_consensus.cc:729] T cbde07c7e0cf4af2b17b16ca92b1847f P 60ea964e26264206b5841a1e1deb3d9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60ea964e26264206b5841a1e1deb3d9d, State: Initialized, Role: FOLLOWER
I20260929 17:56:59.787261 18997 raft_consensus.cc:729] T cbde07c7e0cf4af2b17b16ca92b1847f P 1af4cbe8231e4dcfbe9d2d9effe611b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1af4cbe8231e4dcfbe9d2d9effe611b6, State: Initialized, Role: FOLLOWER
I20260929 17:56:59.787397 18999 consensus_queue.cc:260] T cbde07c7e0cf4af2b17b16ca92b1847f P dd63bce3e264450b87284d5ed110b4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd63bce3e264450b87284d5ed110b4ee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36563 } } peers { permanent_uuid: "60ea964e26264206b5841a1e1deb3d9d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33347 } } peers { permanent_uuid: "1af4cbe8231e4dcfbe9d2d9effe611b6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35193 } }
I20260929 17:56:59.787679 18997 consensus_queue.cc:260] T cbde07c7e0cf4af2b17b16ca92b1847f P 1af4cbe8231e4dcfbe9d2d9effe611b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd63bce3e264450b87284d5ed110b4ee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36563 } } peers { permanent_uuid: "60ea964e26264206b5841a1e1deb3d9d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33347 } } peers { permanent_uuid: "1af4cbe8231e4dcfbe9d2d9effe611b6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35193 } }
I20260929 17:56:59.787679 18998 consensus_queue.cc:260] T cbde07c7e0cf4af2b17b16ca92b1847f P 60ea964e26264206b5841a1e1deb3d9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd63bce3e264450b87284d5ed110b4ee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36563 } } peers { permanent_uuid: "60ea964e26264206b5841a1e1deb3d9d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33347 } } peers { permanent_uuid: "1af4cbe8231e4dcfbe9d2d9effe611b6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35193 } }
I20260929 17:56:59.788125 18980 heartbeater.cc:503] Master 127.0.98.62:38745 was elected leader, sending a full tablet report...
I20260929 17:56:59.788475 18999 ts_tablet_manager.cc:1428] T cbde07c7e0cf4af2b17b16ca92b1847f P dd63bce3e264450b87284d5ed110b4ee: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260929 17:56:59.788633 18997 ts_tablet_manager.cc:1428] T cbde07c7e0cf4af2b17b16ca92b1847f P 1af4cbe8231e4dcfbe9d2d9effe611b6: Time spent starting tablet: real 0.006s user 0.004s sys 0.001s
I20260929 17:56:59.788947 18846 heartbeater.cc:503] Master 127.0.98.62:38745 was elected leader, sending a full tablet report...
I20260929 17:56:59.789359 18712 heartbeater.cc:503] Master 127.0.98.62:38745 was elected leader, sending a full tablet report...
I20260929 17:56:59.789785 18998 ts_tablet_manager.cc:1428] T cbde07c7e0cf4af2b17b16ca92b1847f P 60ea964e26264206b5841a1e1deb3d9d: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
W20260929 17:56:59.801995 18713 tablet.cc:2369] T cbde07c7e0cf4af2b17b16ca92b1847f P 60ea964e26264206b5841a1e1deb3d9d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 17:56:59.963264 19005 raft_consensus.cc:482] T cbde07c7e0cf4af2b17b16ca92b1847f P 60ea964e26264206b5841a1e1deb3d9d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 17:56:59.963503 19005 raft_consensus.cc:504] T cbde07c7e0cf4af2b17b16ca92b1847f P 60ea964e26264206b5841a1e1deb3d9d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd63bce3e264450b87284d5ed110b4ee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36563 } } peers { permanent_uuid: "60ea964e26264206b5841a1e1deb3d9d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33347 } } peers { permanent_uuid: "1af4cbe8231e4dcfbe9d2d9effe611b6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35193 } }
I20260929 17:56:59.964577 19005 leader_election.cc:288] T cbde07c7e0cf4af2b17b16ca92b1847f P 60ea964e26264206b5841a1e1deb3d9d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dd63bce3e264450b87284d5ed110b4ee (127.0.98.3:36563), 1af4cbe8231e4dcfbe9d2d9effe611b6 (127.0.98.2:35193)
W20260929 17:56:59.964627 18981 tablet.cc:2369] T cbde07c7e0cf4af2b17b16ca92b1847f P dd63bce3e264450b87284d5ed110b4ee: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 17:56:59.968801 18996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:56:59.965100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38923 (local address 127.0.98.2:35193)
0929 17:56:59.965215 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0929 17:56:59.965220 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:56:59.965606 (+ 386us) server_negotiation.cc:407] Connection header received
0929 17:56:59.965706 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:56:59.965709 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:56:59.965760 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:56:59.965833 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:56:59.966134 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:56:59.966648 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:56:59.967754 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:56:59.968184 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:56:59.968285 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:56:59.968347 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0929 17:56:59.968582 (+ 235us) server_negotiation.cc:299] Negotiation successful
0929 17:56:59.968659 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 17:56:59.968801 18993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:56:59.965090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43013 (local address 127.0.98.3:36563)
0929 17:56:59.965215 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0929 17:56:59.965219 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:56:59.965467 (+ 248us) server_negotiation.cc:407] Connection header received
0929 17:56:59.965597 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:56:59.965600 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:56:59.965651 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:56:59.965723 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:56:59.966016 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:56:59.966570 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:56:59.967743 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:56:59.968173 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:56:59.968275 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:56:59.968351 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0929 17:56:59.968585 (+ 234us) server_negotiation.cc:299] Negotiation successful
0929 17:56:59.968660 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 17:56:59.968823 19007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:56:59.964872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35193 (local address 127.0.98.1:38923)
0929 17:56:59.965493 (+ 621us) negotiation.cc:107] Waiting for socket to connect
0929 17:56:59.965514 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 17:56:59.965618 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:56:59.965854 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:56:59.965856 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:56:59.965870 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:56:59.966000 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:56:59.966007 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:56:59.966761 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:56:59.966764 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:56:59.967615 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:56:59.967625 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:56:59.968308 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:56:59.968312 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:56:59.968444 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:56:59.968475 (+ 31us) client_negotiation.cc:769] Sending connection context
0929 17:56:59.968550 (+ 75us) client_negotiation.cc:240] Negotiation successful
0929 17:56:59.968637 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":555,"mutex_wait_us":334,"thread_start_us":116,"threads_started":1}
I20260929 17:56:59.968825 19006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:56:59.964872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36563 (local address 127.0.98.1:43013)
0929 17:56:59.965369 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0929 17:56:59.965380 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 17:56:59.965496 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:56:59.965743 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:56:59.965746 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:56:59.965761 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:56:59.965905 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:56:59.965913 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:56:59.966700 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:56:59.966704 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:56:59.967610 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:56:59.967622 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:56:59.968307 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:56:59.968310 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:56:59.968468 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:56:59.968487 (+ 19us) client_negotiation.cc:769] Sending connection context
0929 17:56:59.968563 (+ 76us) client_negotiation.cc:240] Negotiation successful
0929 17:56:59.968642 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":426,"mutex_wait_us":56,"thread_start_us":146,"threads_started":1}
I20260929 17:56:59.969686 18800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbde07c7e0cf4af2b17b16ca92b1847f" candidate_uuid: "60ea964e26264206b5841a1e1deb3d9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1af4cbe8231e4dcfbe9d2d9effe611b6" is_pre_election: true
I20260929 17:56:59.969969 18934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbde07c7e0cf4af2b17b16ca92b1847f" candidate_uuid: "60ea964e26264206b5841a1e1deb3d9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd63bce3e264450b87284d5ed110b4ee" is_pre_election: true
I20260929 17:56:59.970010 18800 raft_consensus.cc:2454] T cbde07c7e0cf4af2b17b16ca92b1847f P 1af4cbe8231e4dcfbe9d2d9effe611b6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60ea964e26264206b5841a1e1deb3d9d in term 0.
I20260929 17:56:59.970252 18934 raft_consensus.cc:2454] T cbde07c7e0cf4af2b17b16ca92b1847f P dd63bce3e264450b87284d5ed110b4ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60ea964e26264206b5841a1e1deb3d9d in term 0.
I20260929 17:56:59.970614 18601 leader_election.cc:302] T cbde07c7e0cf4af2b17b16ca92b1847f P 60ea964e26264206b5841a1e1deb3d9d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1af4cbe8231e4dcfbe9d2d9effe611b6, 60ea964e26264206b5841a1e1deb3d9d; no voters:
I20260929 17:56:59.970892 19005 raft_consensus.cc:2781] T cbde07c7e0cf4af2b17b16ca92b1847f P 60ea964e26264206b5841a1e1deb3d9d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 17:56:59.970983 19005 raft_consensus.cc:482] T cbde07c7e0cf4af2b17b16ca92b1847f P 60ea964e26264206b5841a1e1deb3d9d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 17:56:59.971060 19005 raft_consensus.cc:3037] T cbde07c7e0cf4af2b17b16ca92b1847f P 60ea964e26264206b5841a1e1deb3d9d [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:56:59.972116 19005 raft_consensus.cc:504] T cbde07c7e0cf4af2b17b16ca92b1847f P 60ea964e26264206b5841a1e1deb3d9d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd63bce3e264450b87284d5ed110b4ee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36563 } } peers { permanent_uuid: "60ea964e26264206b5841a1e1deb3d9d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33347 } } peers { permanent_uuid: "1af4cbe8231e4dcfbe9d2d9effe611b6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35193 } }
I20260929 17:56:59.972468 19005 leader_election.cc:288] T cbde07c7e0cf4af2b17b16ca92b1847f P 60ea964e26264206b5841a1e1deb3d9d [CANDIDATE]: Term 1 election: Requested vote from peers dd63bce3e264450b87284d5ed110b4ee (127.0.98.3:36563), 1af4cbe8231e4dcfbe9d2d9effe611b6 (127.0.98.2:35193)
I20260929 17:56:59.972901 18800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbde07c7e0cf4af2b17b16ca92b1847f" candidate_uuid: "60ea964e26264206b5841a1e1deb3d9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1af4cbe8231e4dcfbe9d2d9effe611b6"
I20260929 17:56:59.972900 18934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbde07c7e0cf4af2b17b16ca92b1847f" candidate_uuid: "60ea964e26264206b5841a1e1deb3d9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd63bce3e264450b87284d5ed110b4ee"
I20260929 17:56:59.973050 18800 raft_consensus.cc:3037] T cbde07c7e0cf4af2b17b16ca92b1847f P 1af4cbe8231e4dcfbe9d2d9effe611b6 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:56:59.973075 18934 raft_consensus.cc:3037] T cbde07c7e0cf4af2b17b16ca92b1847f P dd63bce3e264450b87284d5ed110b4ee [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:56:59.974146 18800 raft_consensus.cc:2454] T cbde07c7e0cf4af2b17b16ca92b1847f P 1af4cbe8231e4dcfbe9d2d9effe611b6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60ea964e26264206b5841a1e1deb3d9d in term 1.
I20260929 17:56:59.974148 18934 raft_consensus.cc:2454] T cbde07c7e0cf4af2b17b16ca92b1847f P dd63bce3e264450b87284d5ed110b4ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60ea964e26264206b5841a1e1deb3d9d in term 1.
I20260929 17:56:59.974466 18599 leader_election.cc:302] T cbde07c7e0cf4af2b17b16ca92b1847f P 60ea964e26264206b5841a1e1deb3d9d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60ea964e26264206b5841a1e1deb3d9d, dd63bce3e264450b87284d5ed110b4ee; no voters:
I20260929 17:56:59.974623 19005 raft_consensus.cc:2781] T cbde07c7e0cf4af2b17b16ca92b1847f P 60ea964e26264206b5841a1e1deb3d9d [term 1 FOLLOWER]: Leader election won for term 1
I20260929 17:56:59.974843 19005 raft_consensus.cc:686] T cbde07c7e0cf4af2b17b16ca92b1847f P 60ea964e26264206b5841a1e1deb3d9d [term 1 LEADER]: Becoming Leader. State: Replica: 60ea964e26264206b5841a1e1deb3d9d, State: Running, Role: LEADER
I20260929 17:56:59.975191 19005 consensus_queue.cc:237] T cbde07c7e0cf4af2b17b16ca92b1847f P 60ea964e26264206b5841a1e1deb3d9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd63bce3e264450b87284d5ed110b4ee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36563 } } peers { permanent_uuid: "60ea964e26264206b5841a1e1deb3d9d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33347 } } peers { permanent_uuid: "1af4cbe8231e4dcfbe9d2d9effe611b6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35193 } }
I20260929 17:56:59.978369 18521 catalog_manager.cc:5495] T cbde07c7e0cf4af2b17b16ca92b1847f P 60ea964e26264206b5841a1e1deb3d9d reported cstate change: term changed from 0 to 1, leader changed from <none> to 60ea964e26264206b5841a1e1deb3d9d (127.0.98.1). New cstate: current_term: 1 leader_uuid: "60ea964e26264206b5841a1e1deb3d9d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd63bce3e264450b87284d5ed110b4ee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36563 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "60ea964e26264206b5841a1e1deb3d9d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33347 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1af4cbe8231e4dcfbe9d2d9effe611b6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35193 } health_report { overall_health: UNKNOWN } } }
W20260929 17:57:00.008111 18847 tablet.cc:2369] T cbde07c7e0cf4af2b17b16ca92b1847f P 1af4cbe8231e4dcfbe9d2d9effe611b6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 17:57:00.017174 18994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:00.012908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58454 (local address 127.0.98.1:33347)
0929 17:57:00.013067 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0929 17:57:00.013079 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 17:57:00.013095 (+ 16us) server_negotiation.cc:407] Connection header received
0929 17:57:00.013238 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:57:00.013241 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:57:00.013308 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:57:00.013433 (+ 125us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 17:57:00.013682 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:00.014436 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:00.015328 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:00.015542 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:00.015631 (+ 89us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:00.015927 (+ 296us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 17:57:00.016364 (+ 437us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 17:57:00.016633 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0929 17:57:00.016961 (+ 328us) server_negotiation.cc:299] Negotiation successful
0929 17:57:00.017035 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 17:57:00.023245 18934 raft_consensus.cc:1261] T cbde07c7e0cf4af2b17b16ca92b1847f P dd63bce3e264450b87284d5ed110b4ee [term 1 FOLLOWER]: Refusing update from remote peer 60ea964e26264206b5841a1e1deb3d9d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 17:57:00.023258 18800 raft_consensus.cc:1261] T cbde07c7e0cf4af2b17b16ca92b1847f P 1af4cbe8231e4dcfbe9d2d9effe611b6 [term 1 FOLLOWER]: Refusing update from remote peer 60ea964e26264206b5841a1e1deb3d9d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 17:57:00.023903 19005 consensus_queue.cc:1035] T cbde07c7e0cf4af2b17b16ca92b1847f P 60ea964e26264206b5841a1e1deb3d9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd63bce3e264450b87284d5ed110b4ee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 17:57:00.024077 19008 consensus_queue.cc:1035] T cbde07c7e0cf4af2b17b16ca92b1847f P 60ea964e26264206b5841a1e1deb3d9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1af4cbe8231e4dcfbe9d2d9effe611b6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 17:57:00.032091 19013 mvcc.cc:205] Tried to move back new op lower bound from 7334726123608076288 to 7334726123421048832. Current Snapshot: MvccSnapshot[applied={T|T < 7334726123608076288}]
I20260929 17:57:00.033437 19012 mvcc.cc:205] Tried to move back new op lower bound from 7334726123608076288 to 7334726123421048832. Current Snapshot: MvccSnapshot[applied={T|T < 7334726123608076288}]
I20260929 17:57:00.039219 18520 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33286:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 17:57:00.039407 18520 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33286:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 17:57:00.041868 18520 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 092f892a1adf4a4793bfe28230dd9611: Sending DeleteTablet for 3 replicas of tablet cbde07c7e0cf4af2b17b16ca92b1847f
I20260929 17:57:00.042675 18646 tablet_service.cc:1514] Processing DeleteTablet for tablet cbde07c7e0cf4af2b17b16ca92b1847f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 17:57:00 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58440
I20260929 17:57:00.042661 18914 tablet_service.cc:1514] Processing DeleteTablet for tablet cbde07c7e0cf4af2b17b16ca92b1847f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 17:57:00 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33876
I20260929 17:57:00.042762 18780 tablet_service.cc:1514] Processing DeleteTablet for tablet cbde07c7e0cf4af2b17b16ca92b1847f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 17:57:00 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56636
I20260929 17:57:00.043270 19022 tablet_replica.cc:330] T cbde07c7e0cf4af2b17b16ca92b1847f P 1af4cbe8231e4dcfbe9d2d9effe611b6: stopping tablet replica
I20260929 17:57:00.043490 19020 tablet_replica.cc:330] T cbde07c7e0cf4af2b17b16ca92b1847f P dd63bce3e264450b87284d5ed110b4ee: stopping tablet replica
I20260929 17:57:00.043602 19022 raft_consensus.cc:2229] T cbde07c7e0cf4af2b17b16ca92b1847f P 1af4cbe8231e4dcfbe9d2d9effe611b6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 17:57:00.043886 19020 raft_consensus.cc:2229] T cbde07c7e0cf4af2b17b16ca92b1847f P dd63bce3e264450b87284d5ed110b4ee [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 17:57:00.043934 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskup3Soe/build/debug/bin/kudu with pid 18583
I20260929 17:57:00.043937 19022 raft_consensus.cc:2258] T cbde07c7e0cf4af2b17b16ca92b1847f P 1af4cbe8231e4dcfbe9d2d9effe611b6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 17:57:00.044864 19020 raft_consensus.cc:2258] T cbde07c7e0cf4af2b17b16ca92b1847f P dd63bce3e264450b87284d5ed110b4ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 17:57:00.046473 19020 ts_tablet_manager.cc:1905] T cbde07c7e0cf4af2b17b16ca92b1847f P dd63bce3e264450b87284d5ed110b4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 17:57:00.046473 19022 ts_tablet_manager.cc:1905] T cbde07c7e0cf4af2b17b16ca92b1847f P 1af4cbe8231e4dcfbe9d2d9effe611b6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 17:57:00.048830 19020 ts_tablet_manager.cc:1918] T cbde07c7e0cf4af2b17b16ca92b1847f P dd63bce3e264450b87284d5ed110b4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 17:57:00.048830 19022 ts_tablet_manager.cc:1918] T cbde07c7e0cf4af2b17b16ca92b1847f P 1af4cbe8231e4dcfbe9d2d9effe611b6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 17:57:00.048945 19022 log.cc:1196] T cbde07c7e0cf4af2b17b16ca92b1847f P 1af4cbe8231e4dcfbe9d2d9effe611b6: Deleting WAL directory at /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704375975685-392-0/minicluster-data/ts-1/wal/wals/cbde07c7e0cf4af2b17b16ca92b1847f
I20260929 17:57:00.048949 19020 log.cc:1196] T cbde07c7e0cf4af2b17b16ca92b1847f P dd63bce3e264450b87284d5ed110b4ee: Deleting WAL directory at /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790704375975685-392-0/minicluster-data/ts-2/wal/wals/cbde07c7e0cf4af2b17b16ca92b1847f
I20260929 17:57:00.049314 19022 ts_tablet_manager.cc:1939] T cbde07c7e0cf4af2b17b16ca92b1847f P 1af4cbe8231e4dcfbe9d2d9effe611b6: Deleting consensus metadata
I20260929 17:57:00.049314 19020 ts_tablet_manager.cc:1939] T cbde07c7e0cf4af2b17b16ca92b1847f P dd63bce3e264450b87284d5ed110b4ee: Deleting consensus metadata
I20260929 17:57:00.050751 18505 catalog_manager.cc:4841] TS dd63bce3e264450b87284d5ed110b4ee (127.0.98.3:36563): tablet cbde07c7e0cf4af2b17b16ca92b1847f (table test-table [id=9f6d05fc01e546ab9cfcc0be4e609313]) successfully deleted
I20260929 17:57:00.050751 18507 catalog_manager.cc:4841] TS 1af4cbe8231e4dcfbe9d2d9effe611b6 (127.0.98.2:35193): tablet cbde07c7e0cf4af2b17b16ca92b1847f (table test-table [id=9f6d05fc01e546ab9cfcc0be4e609313]) successfully deleted
W20260929 17:57:00.052649 18507 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:33347 (error 108)
I20260929 17:57:00.052743 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskup3Soe/build/debug/bin/kudu with pid 18717
I20260929 17:57:00.054209 18992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:00.053852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33347 (local address 127.0.0.1:58470)
0929 17:57:00.054058 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0929 17:57:00.054130 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33347: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121}
W20260929 17:57:00.054374 18507 catalog_manager.cc:4584] TS 60ea964e26264206b5841a1e1deb3d9d (127.0.98.1:33347): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet cbde07c7e0cf4af2b17b16ca92b1847f: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33347: connect: Connection refused (error 111)
I20260929 17:57:00.057890 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskup3Soe/build/debug/bin/kudu with pid 18851
I20260929 17:57:00.062628 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskup3Soe/build/debug/bin/kudu with pid 18489
2026-09-29T17:57:00Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2329 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-29T17:57:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T17:57:00Z Disabled control of system clock
I20260929 17:57:00.090837 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskup3Soe/build/debug/bin/kudu
/tmp/dist-test-taskup3Soe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790704375975685-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790704375975685-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790704375975685-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790704375975685-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39937
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:34013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39937
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 17:57:00.200569 19027 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:57:00.200853 19027 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:57:00.200889 19027 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:57:00.205413 19027 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 17:57:00.205490 19027 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:57:00.205505 19027 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:57:00.205520 19027 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 17:57:00.205534 19027 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 17:57:00.210577 19027 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790704375975685-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790704375975685-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39937
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39937
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790704375975685-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19027
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790704375975685-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:57:00.211727 19027 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:57:00.213179 19027 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:57:00.221590 19032 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:57:00.221649 19035 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:57:00.221602 19033 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:57:00.221838 19027 server_base.cc:1034] running on GCE node
I20260929 17:57:00.222568 19027 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:57:00.223724 19027 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:57:00.224913 19027 hybrid_clock.cc:648] HybridClock initialized: now 1790704620224897 us; error 29 us; skew 500 ppm
I20260929 17:57:00.226909 19027 webserver.cc:460] Webserver started at http://127.0.98.62:38269/ using document root <none> and password file <none>
I20260929 17:57:00.227471 19027 fs_manager.cc:359] Metadata directory not provided
I20260929 17:57:00.227540 19027 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:57:00.227726 19027 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:57:00.230319 19027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790704375975685-392-0/minicluster-data/master-0/data/instance:
uuid: "5539e7f336684188a21ea0d971351f58"
format_stamp: "Formatted at 2026-09-29 17:57:00 on dist-test-slave-53nh"
server_key: "2c0020f77f4c0062cb8abf2c0dd4791b"
server_key_iv: "e56cad4bc59c407074edb0add79a1fe7"
server_key_version: "encryptionkey@0"
I20260929 17:57:00.231000 19027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790704375975685-392-0/minicluster-data/master-0/wal/instance:
uuid: "5539e7f336684188a21ea0d971351f58"
format_stamp: "Formatted at 2026-09-29 17:57:00 on dist-test-slave-53nh"
server_key: "2c0020f77f4c0062cb8abf2c0dd4791b"
server_key_iv: "e56cad4bc59c407074edb0add79a1fe7"
server_key_version: "encryptionkey@0"
I20260929 17:57:00.234803 19027 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 17:57:00.237761 19041 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:57:00.239178 19027 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 17:57:00.239306 19027 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790704375975685-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790704375975685-392-0/minicluster-data/master-0/wal
uuid: "5539e7f336684188a21ea0d971351f58"
format_stamp: "Formatted at 2026-09-29 17:57:00 on dist-test-slave-53nh"
server_key: "2c0020f77f4c0062cb8abf2c0dd4791b"
server_key_iv: "e56cad4bc59c407074edb0add79a1fe7"
server_key_version: "encryptionkey@0"
I20260929 17:57:00.239434 19027 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790704375975685-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790704375975685-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790704375975685-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:57:00.265092 19027 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:57:00.265857 19027 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:57:00.266031 19027 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:57:00.272424 19027 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39937
I20260929 17:57:00.272456 19093 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39937 every 8 connection(s)
I20260929 17:57:00.273389 19027 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790704375975685-392-0/minicluster-data/master-0/data/info.pb
I20260929 17:57:00.276413 19094 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:57:00.276932 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskup3Soe/build/debug/bin/kudu as pid 19027
I20260929 17:57:00.277074 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790704375975685-392-0/minicluster-data/master-0/wal/instance
I20260929 17:57:00.277390 392 external_mini_cluster.cc:1428] Setting key 062a0add55662a48e1a0950627fe5331
I20260929 17:57:00.282563 19094 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5539e7f336684188a21ea0d971351f58: Bootstrap starting.
I20260929 17:57:00.285097 19094 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5539e7f336684188a21ea0d971351f58: Neither blocks nor log segments found. Creating new log.
I20260929 17:57:00.285919 19094 log.cc:824] T 00000000000000000000000000000000 P 5539e7f336684188a21ea0d971351f58: Log is configured to *not* fsync() on all Append() calls
I20260929 17:57:00.286672 19097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:00.278838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39538 (local address 127.0.98.62:39937)
0929 17:57:00.279458 (+ 620us) server_negotiation.cc:206] Beginning negotiation
0929 17:57:00.279471 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 17:57:00.279505 (+ 34us) server_negotiation.cc:407] Connection header received
0929 17:57:00.280181 (+ 676us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:57:00.280197 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:57:00.280596 (+ 399us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:57:00.280868 (+ 272us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:57:00.281270 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:00.282201 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:00.283326 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:00.283817 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:00.283929 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:00.284737 (+ 808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:57:00.284743 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:57:00.284755 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 17:57:00.285228 (+ 473us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:57:00.285284 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 17:57:00.285305 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:57:00.285377 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0929 17:57:00.285515 (+ 138us) server_negotiation.cc:299] Negotiation successful
0929 17:57:00.285780 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274,"thread_start_us":121,"threads_started":1}
I20260929 17:57:00.288182 19094 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5539e7f336684188a21ea0d971351f58: No bootstrap required, opened a new log
I20260929 17:57:00.291765 19094 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5539e7f336684188a21ea0d971351f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5539e7f336684188a21ea0d971351f58" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39937 } }
I20260929 17:57:00.291996 19094 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5539e7f336684188a21ea0d971351f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:57:00.292061 19094 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5539e7f336684188a21ea0d971351f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5539e7f336684188a21ea0d971351f58, State: Initialized, Role: FOLLOWER
I20260929 17:57:00.292500 19094 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5539e7f336684188a21ea0d971351f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5539e7f336684188a21ea0d971351f58" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39937 } }
I20260929 17:57:00.292660 19094 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5539e7f336684188a21ea0d971351f58 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 17:57:00.292722 19094 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5539e7f336684188a21ea0d971351f58 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 17:57:00.292824 19094 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5539e7f336684188a21ea0d971351f58 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:57:00.293785 19094 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5539e7f336684188a21ea0d971351f58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5539e7f336684188a21ea0d971351f58" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39937 } }
I20260929 17:57:00.294117 19094 leader_election.cc:302] T 00000000000000000000000000000000 P 5539e7f336684188a21ea0d971351f58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5539e7f336684188a21ea0d971351f58; no voters:
I20260929 17:57:00.294404 19094 leader_election.cc:288] T 00000000000000000000000000000000 P 5539e7f336684188a21ea0d971351f58 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 17:57:00.294555 19099 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5539e7f336684188a21ea0d971351f58 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 17:57:00.294792 19099 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5539e7f336684188a21ea0d971351f58 [term 1 LEADER]: Becoming Leader. State: Replica: 5539e7f336684188a21ea0d971351f58, State: Running, Role: LEADER
I20260929 17:57:00.295125 19099 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5539e7f336684188a21ea0d971351f58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5539e7f336684188a21ea0d971351f58" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39937 } }
I20260929 17:57:00.295680 19094 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5539e7f336684188a21ea0d971351f58 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 17:57:00.296840 19100 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5539e7f336684188a21ea0d971351f58 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5539e7f336684188a21ea0d971351f58" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5539e7f336684188a21ea0d971351f58" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39937 } } }
I20260929 17:57:00.296974 19100 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5539e7f336684188a21ea0d971351f58 [sys.catalog]: This master's current role is: LEADER
I20260929 17:57:00.297214 19101 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5539e7f336684188a21ea0d971351f58 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5539e7f336684188a21ea0d971351f58. Latest consensus state: current_term: 1 leader_uuid: "5539e7f336684188a21ea0d971351f58" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5539e7f336684188a21ea0d971351f58" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39937 } } }
I20260929 17:57:00.297379 19101 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5539e7f336684188a21ea0d971351f58 [sys.catalog]: This master's current role is: LEADER
I20260929 17:57:00.297397 19108 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 17:57:00.300473 19108 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 17:57:00.305795 19108 catalog_manager.cc:1347] Generated new cluster ID: 64d6c55d50664a01a752466f644716c8
I20260929 17:57:00.305872 19108 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 17:57:00.317795 19108 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 17:57:00.319095 19108 catalog_manager.cc:1504] Loading token signing keys...
I20260929 17:57:00.333050 19108 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5539e7f336684188a21ea0d971351f58: Generated new TSK 0
I20260929 17:57:00.333967 19108 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 17:57:00.345593 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskup3Soe/build/debug/bin/kudu
/tmp/dist-test-taskup3Soe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790704375975685-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790704375975685-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790704375975685-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790704375975685-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39937
--builtin_ntp_servers=127.0.98.20:34013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 17:57:00.457852 19118 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:57:00.458164 19118 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:57:00.458204 19118 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:57:00.462476 19118 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:57:00.462539 19118 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:57:00.462598 19118 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 17:57:00.467621 19118 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790704375975685-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790704375975685-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790704375975685-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39937
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19118
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790704375975685-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:57:00.468776 19118 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:57:00.470275 19118 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:57:00.479991 19126 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:57:00.480013 19123 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:57:00.480014 19124 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:57:00.480252 19118 server_base.cc:1034] running on GCE node
I20260929 17:57:00.480767 19118 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:57:00.481484 19118 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:57:00.482707 19118 hybrid_clock.cc:648] HybridClock initialized: now 1790704620482677 us; error 42 us; skew 500 ppm
I20260929 17:57:00.484536 19118 webserver.cc:460] Webserver started at http://127.0.98.1:33239/ using document root <none> and password file <none>
I20260929 17:57:00.485214 19118 fs_manager.cc:359] Metadata directory not provided
I20260929 17:57:00.485287 19118 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:57:00.485502 19118 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:57:00.487743 19118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790704375975685-392-0/minicluster-data/ts-0/data/instance:
uuid: "bce413dfb85f4434826050803219ca0a"
format_stamp: "Formatted at 2026-09-29 17:57:00 on dist-test-slave-53nh"
server_key: "94047b484f24e69dec93291cf1421bb1"
server_key_iv: "2078791ecbb3e09120bbd482a8f20462"
server_key_version: "encryptionkey@0"
I20260929 17:57:00.488358 19118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790704375975685-392-0/minicluster-data/ts-0/wal/instance:
uuid: "bce413dfb85f4434826050803219ca0a"
format_stamp: "Formatted at 2026-09-29 17:57:00 on dist-test-slave-53nh"
server_key: "94047b484f24e69dec93291cf1421bb1"
server_key_iv: "2078791ecbb3e09120bbd482a8f20462"
server_key_version: "encryptionkey@0"
I20260929 17:57:00.492501 19118 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 17:57:00.495298 19132 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:57:00.496840 19118 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 17:57:00.496956 19118 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790704375975685-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790704375975685-392-0/minicluster-data/ts-0/wal
uuid: "bce413dfb85f4434826050803219ca0a"
format_stamp: "Formatted at 2026-09-29 17:57:00 on dist-test-slave-53nh"
server_key: "94047b484f24e69dec93291cf1421bb1"
server_key_iv: "2078791ecbb3e09120bbd482a8f20462"
server_key_version: "encryptionkey@0"
I20260929 17:57:00.497124 19118 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790704375975685-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790704375975685-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790704375975685-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:57:00.508971 19118 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:57:00.509701 19118 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:57:00.509867 19118 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:57:00.510502 19118 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 17:57:00.511456 19118 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:57:00.511502 19118 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:57:00.511545 19118 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:57:00.511574 19118 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 17:57:00.519927 19118 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46303
I20260929 17:57:00.519945 19245 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46303 every 8 connection(s)
I20260929 17:57:00.520850 19118 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790704375975685-392-0/minicluster-data/ts-0/data/info.pb
I20260929 17:57:00.521698 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskup3Soe/build/debug/bin/kudu as pid 19118
I20260929 17:57:00.521817 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790704375975685-392-0/minicluster-data/ts-0/wal/instance
I20260929 17:57:00.522082 392 external_mini_cluster.cc:1428] Setting key be2e5162650eccb7c6b90336db68319b
I20260929 17:57:00.524173 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskup3Soe/build/debug/bin/kudu
/tmp/dist-test-taskup3Soe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790704375975685-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790704375975685-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790704375975685-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790704375975685-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39937
--builtin_ntp_servers=127.0.98.20:34013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 17:57:00.531944 19097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:00.522893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37203 (local address 127.0.98.62:39937)
0929 17:57:00.523034 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0929 17:57:00.523039 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:57:00.524157 (+ 1118us) server_negotiation.cc:407] Connection header received
0929 17:57:00.525812 (+ 1655us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:57:00.525815 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:57:00.525865 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:57:00.525943 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:57:00.527213 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:00.527821 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:00.529663 (+ 1842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:00.529909 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:00.529997 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:00.531431 (+ 1434us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:57:00.531434 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:57:00.531436 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 17:57:00.531451 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:57:00.531479 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 17:57:00.531488 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:57:00.531561 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0929 17:57:00.531751 (+ 190us) server_negotiation.cc:299] Negotiation successful
0929 17:57:00.531818 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 17:57:00.533298 19248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:00.523320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39937 (local address 127.0.98.1:37203)
0929 17:57:00.524009 (+ 689us) negotiation.cc:107] Waiting for socket to connect
0929 17:57:00.524052 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 17:57:00.525550 (+ 1498us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:57:00.526166 (+ 616us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:57:00.526188 (+ 22us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:57:00.526313 (+ 125us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:57:00.527031 (+ 718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:57:00.527053 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:57:00.527985 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:57:00.527989 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:57:00.529506 (+ 1517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:57:00.529520 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:57:00.530088 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:57:00.530093 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:57:00.530326 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:00.531040 (+ 714us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 17:57:00.531042 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:57:00.531111 (+ 69us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 17:57:00.531213 (+ 102us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:57:00.531583 (+ 370us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:57:00.531593 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:57:00.531600 (+ 7us) client_negotiation.cc:769] Sending connection context
0929 17:57:00.531760 (+ 160us) client_negotiation.cc:240] Negotiation successful
0929 17:57:00.532126 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":124,"threads_started":1}
I20260929 17:57:00.534725 19246 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39937
I20260929 17:57:00.535027 19246 heartbeater.cc:464] Registering TS with master...
I20260929 17:57:00.535748 19246 heartbeater.cc:511] Master 127.0.98.62:39937 requested a full tablet report, sending...
I20260929 17:57:00.537561 19057 ts_manager.cc:194] Registered new tserver with Master: bce413dfb85f4434826050803219ca0a (127.0.98.1:46303)
I20260929 17:57:00.538887 19057 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:37203
W20260929 17:57:00.639987 19249 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:57:00.640283 19249 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:57:00.640344 19249 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:57:00.644634 19249 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:57:00.644712 19249 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:57:00.644799 19249 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 17:57:00.649868 19249 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790704375975685-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790704375975685-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790704375975685-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39937
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19249
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790704375975685-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:57:00.651062 19249 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:57:00.652478 19249 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:57:00.662323 19257 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:57:00.662343 19255 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:57:00.662334 19254 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:57:00.662374 19249 server_base.cc:1034] running on GCE node
I20260929 17:57:00.663012 19249 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:57:00.663606 19249 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:57:00.664786 19249 hybrid_clock.cc:648] HybridClock initialized: now 1790704620664757 us; error 40 us; skew 500 ppm
I20260929 17:57:00.666838 19249 webserver.cc:460] Webserver started at http://127.0.98.2:38299/ using document root <none> and password file <none>
I20260929 17:57:00.667413 19249 fs_manager.cc:359] Metadata directory not provided
I20260929 17:57:00.667478 19249 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:57:00.667698 19249 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:57:00.669826 19249 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790704375975685-392-0/minicluster-data/ts-1/data/instance:
uuid: "8c67995864854c3eaf6867744cbc9c0f"
format_stamp: "Formatted at 2026-09-29 17:57:00 on dist-test-slave-53nh"
server_key: "c281e8bb7a0b6aa12c84d06eeb2e1d52"
server_key_iv: "5ee08e93d22bc0f7bd1e8b973d932a62"
server_key_version: "encryptionkey@0"
I20260929 17:57:00.670364 19249 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790704375975685-392-0/minicluster-data/ts-1/wal/instance:
uuid: "8c67995864854c3eaf6867744cbc9c0f"
format_stamp: "Formatted at 2026-09-29 17:57:00 on dist-test-slave-53nh"
server_key: "c281e8bb7a0b6aa12c84d06eeb2e1d52"
server_key_iv: "5ee08e93d22bc0f7bd1e8b973d932a62"
server_key_version: "encryptionkey@0"
I20260929 17:57:00.674109 19249 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 17:57:00.676723 19263 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:57:00.678031 19249 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 17:57:00.678150 19249 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790704375975685-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790704375975685-392-0/minicluster-data/ts-1/wal
uuid: "8c67995864854c3eaf6867744cbc9c0f"
format_stamp: "Formatted at 2026-09-29 17:57:00 on dist-test-slave-53nh"
server_key: "c281e8bb7a0b6aa12c84d06eeb2e1d52"
server_key_iv: "5ee08e93d22bc0f7bd1e8b973d932a62"
server_key_version: "encryptionkey@0"
I20260929 17:57:00.678256 19249 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790704375975685-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790704375975685-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790704375975685-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:57:00.690802 19249 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:57:00.691517 19249 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:57:00.691697 19249 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:57:00.692276 19249 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 17:57:00.693255 19249 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:57:00.693305 19249 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:57:00.693357 19249 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:57:00.693380 19249 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 17:57:00.702117 19249 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35217
I20260929 17:57:00.702153 19376 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35217 every 8 connection(s)
I20260929 17:57:00.703019 19249 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790704375975685-392-0/minicluster-data/ts-1/data/info.pb
I20260929 17:57:00.710829 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskup3Soe/build/debug/bin/kudu as pid 19249
I20260929 17:57:00.710938 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790704375975685-392-0/minicluster-data/ts-1/wal/instance
I20260929 17:57:00.711259 392 external_mini_cluster.cc:1428] Setting key e8abc2915021408b06aefa44c1043778
I20260929 17:57:00.711910 19097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:00.704782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40269 (local address 127.0.98.62:39937)
0929 17:57:00.704957 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0929 17:57:00.704962 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:57:00.706076 (+ 1114us) server_negotiation.cc:407] Connection header received
0929 17:57:00.707419 (+ 1343us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:57:00.707422 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:57:00.707471 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:57:00.707545 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:57:00.708400 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:00.708916 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:00.710026 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:00.710244 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:00.710315 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:00.711424 (+ 1109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:57:00.711428 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:57:00.711430 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 17:57:00.711442 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:57:00.711462 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 17:57:00.711471 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:57:00.711528 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0929 17:57:00.711710 (+ 182us) server_negotiation.cc:299] Negotiation successful
0929 17:57:00.711776 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 17:57:00.712860 19379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:00.705263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39937 (local address 127.0.98.2:40269)
0929 17:57:00.705922 (+ 659us) negotiation.cc:107] Waiting for socket to connect
0929 17:57:00.705963 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 17:57:00.707222 (+ 1259us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:57:00.707694 (+ 472us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:57:00.707700 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:57:00.707784 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:57:00.708291 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:57:00.708303 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:57:00.709037 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:57:00.709040 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:57:00.709933 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:57:00.709942 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:57:00.710353 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:57:00.710356 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:57:00.710492 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:00.711111 (+ 619us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 17:57:00.711114 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:57:00.711169 (+ 55us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 17:57:00.711250 (+ 81us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:57:00.711550 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:57:00.711558 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:57:00.711564 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 17:57:00.711688 (+ 124us) client_negotiation.cc:240] Negotiation successful
0929 17:57:00.711945 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":115,"threads_started":1}
I20260929 17:57:00.713368 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskup3Soe/build/debug/bin/kudu
/tmp/dist-test-taskup3Soe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790704375975685-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790704375975685-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790704375975685-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790704375975685-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39937
--builtin_ntp_servers=127.0.98.20:34013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 17:57:00.713968 19377 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39937
I20260929 17:57:00.714205 19377 heartbeater.cc:464] Registering TS with master...
I20260929 17:57:00.714768 19377 heartbeater.cc:511] Master 127.0.98.62:39937 requested a full tablet report, sending...
I20260929 17:57:00.715731 19057 ts_manager.cc:194] Registered new tserver with Master: 8c67995864854c3eaf6867744cbc9c0f (127.0.98.2:35217)
I20260929 17:57:00.716264 19057 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:40269
W20260929 17:57:00.824234 19380 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:57:00.824522 19380 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:57:00.824577 19380 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:57:00.828914 19380 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:57:00.828989 19380 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:57:00.829092 19380 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 17:57:00.834009 19380 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790704375975685-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790704375975685-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790704375975685-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39937
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19380
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790704375975685-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:57:00.835070 19380 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:57:00.836418 19380 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:57:00.846112 19386 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:57:00.846207 19380 server_base.cc:1034] running on GCE node
W20260929 17:57:00.846095 19385 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:57:00.846092 19388 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:57:00.846837 19380 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:57:00.847558 19380 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:57:00.848737 19380 hybrid_clock.cc:648] HybridClock initialized: now 1790704620848705 us; error 49 us; skew 500 ppm
I20260929 17:57:00.850850 19380 webserver.cc:460] Webserver started at http://127.0.98.3:38587/ using document root <none> and password file <none>
I20260929 17:57:00.851486 19380 fs_manager.cc:359] Metadata directory not provided
I20260929 17:57:00.851553 19380 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:57:00.851758 19380 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:57:00.854185 19380 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790704375975685-392-0/minicluster-data/ts-2/data/instance:
uuid: "bd0a0ea4e780444da9109e10a6313db0"
format_stamp: "Formatted at 2026-09-29 17:57:00 on dist-test-slave-53nh"
server_key: "696a2ddf20a3dd5365e71aa395ca86a3"
server_key_iv: "e61f211440da94bb8881b2f0b65bcc4b"
server_key_version: "encryptionkey@0"
I20260929 17:57:00.854828 19380 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790704375975685-392-0/minicluster-data/ts-2/wal/instance:
uuid: "bd0a0ea4e780444da9109e10a6313db0"
format_stamp: "Formatted at 2026-09-29 17:57:00 on dist-test-slave-53nh"
server_key: "696a2ddf20a3dd5365e71aa395ca86a3"
server_key_iv: "e61f211440da94bb8881b2f0b65bcc4b"
server_key_version: "encryptionkey@0"
I20260929 17:57:00.859033 19380 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 17:57:00.861720 19394 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:57:00.863034 19380 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 17:57:00.863152 19380 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790704375975685-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790704375975685-392-0/minicluster-data/ts-2/wal
uuid: "bd0a0ea4e780444da9109e10a6313db0"
format_stamp: "Formatted at 2026-09-29 17:57:00 on dist-test-slave-53nh"
server_key: "696a2ddf20a3dd5365e71aa395ca86a3"
server_key_iv: "e61f211440da94bb8881b2f0b65bcc4b"
server_key_version: "encryptionkey@0"
I20260929 17:57:00.863269 19380 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790704375975685-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790704375975685-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790704375975685-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:57:00.890946 19380 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:57:00.891739 19380 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:57:00.891923 19380 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:57:00.892551 19380 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 17:57:00.893574 19380 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:57:00.893622 19380 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:57:00.893666 19380 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:57:00.893697 19380 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 17:57:00.902238 19380 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40635
I20260929 17:57:00.902264 19507 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40635 every 8 connection(s)
I20260929 17:57:00.903138 19380 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790704375975685-392-0/minicluster-data/ts-2/data/info.pb
I20260929 17:57:00.909821 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskup3Soe/build/debug/bin/kudu as pid 19380
I20260929 17:57:00.909904 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790704375975685-392-0/minicluster-data/ts-2/wal/instance
I20260929 17:57:00.910153 392 external_mini_cluster.cc:1428] Setting key 434007f50a89f7794fcd3089bfe0ac89
I20260929 17:57:00.912569 19097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:00.904981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57337 (local address 127.0.98.62:39937)
0929 17:57:00.905154 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0929 17:57:00.905159 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:57:00.906279 (+ 1120us) server_negotiation.cc:407] Connection header received
0929 17:57:00.907663 (+ 1384us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:57:00.907666 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:57:00.907716 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:57:00.907789 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:57:00.908707 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:00.909294 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:00.910618 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:00.910902 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:00.910996 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:00.912152 (+ 1156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:57:00.912155 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:57:00.912158 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 17:57:00.912172 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:57:00.912195 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 17:57:00.912205 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:57:00.912261 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0929 17:57:00.912412 (+ 151us) server_negotiation.cc:299] Negotiation successful
0929 17:57:00.912464 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260929 17:57:00.913479 19510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:00.905444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39937 (local address 127.0.98.3:57337)
0929 17:57:00.906118 (+ 674us) negotiation.cc:107] Waiting for socket to connect
0929 17:57:00.906161 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 17:57:00.907439 (+ 1278us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:57:00.907942 (+ 503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:57:00.907949 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:57:00.908037 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:57:00.908567 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:57:00.908581 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:57:00.909408 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:57:00.909410 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:57:00.910390 (+ 980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:57:00.910401 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:57:00.911066 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:57:00.911070 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:57:00.911236 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:00.911851 (+ 615us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 17:57:00.911853 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:57:00.911907 (+ 54us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 17:57:00.911986 (+ 79us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:57:00.912281 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:57:00.912289 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:57:00.912295 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 17:57:00.912400 (+ 105us) client_negotiation.cc:240] Negotiation successful
0929 17:57:00.912632 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":115,"threads_started":1}
I20260929 17:57:00.914495 19508 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39937
I20260929 17:57:00.914723 19508 heartbeater.cc:464] Registering TS with master...
I20260929 17:57:00.915254 19508 heartbeater.cc:511] Master 127.0.98.62:39937 requested a full tablet report, sending...
I20260929 17:57:00.916263 19057 ts_manager.cc:194] Registered new tserver with Master: bd0a0ea4e780444da9109e10a6313db0 (127.0.98.3:40635)
I20260929 17:57:00.916793 19057 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:57337
I20260929 17:57:00.924150 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 17:57:00.929931 19097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:00.926105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39540 (local address 127.0.98.62:39937)
0929 17:57:00.926229 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0929 17:57:00.926233 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:57:00.926406 (+ 173us) server_negotiation.cc:407] Connection header received
0929 17:57:00.926507 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:57:00.926510 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:57:00.926549 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:57:00.926642 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:57:00.926939 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:00.927437 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:00.928262 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:00.928466 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:00.928542 (+ 76us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:00.929479 (+ 937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:57:00.929482 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:57:00.929484 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 17:57:00.929498 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:57:00.929517 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 17:57:00.929526 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:57:00.929644 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0929 17:57:00.929764 (+ 120us) server_negotiation.cc:299] Negotiation successful
0929 17:57:00.929803 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260929 17:57:00.932874 19057 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39540:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 17:57:00.935958 19057 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 17:57:00.951013 19522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:00.945219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46303 (local address 127.0.0.1:35952)
0929 17:57:00.946010 (+ 791us) negotiation.cc:107] Waiting for socket to connect
0929 17:57:00.946022 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 17:57:00.946103 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:57:00.947468 (+ 1365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:57:00.947471 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:57:00.947490 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:57:00.947563 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:57:00.947568 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:57:00.948740 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:57:00.948744 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:57:00.949769 (+ 1025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:57:00.949777 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:57:00.950507 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:57:00.950510 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:57:00.950610 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:00.950646 (+ 36us) client_negotiation.cc:769] Sending connection context
0929 17:57:00.950739 (+ 93us) client_negotiation.cc:240] Negotiation successful
0929 17:57:00.950851 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":732,"mutex_wait_us":2,"thread_start_us":102,"threads_started":1}
I20260929 17:57:00.951004 19523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:00.945771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35952 (local address 127.0.98.1:46303)
0929 17:57:00.947066 (+ 1295us) server_negotiation.cc:206] Beginning negotiation
0929 17:57:00.947077 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 17:57:00.947098 (+ 21us) server_negotiation.cc:407] Connection header received
0929 17:57:00.947168 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:57:00.947172 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:57:00.947337 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:57:00.947474 (+ 137us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:57:00.947663 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:00.948647 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:00.949873 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:00.950402 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:00.950510 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:00.950712 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0929 17:57:00.950764 (+ 52us) server_negotiation.cc:299] Negotiation successful
0929 17:57:00.950855 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1196,"thread_start_us":142,"threads_started":1}
I20260929 17:57:00.951359 19524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:00.945909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51890 (local address 127.0.98.2:35217)
0929 17:57:00.946283 (+ 374us) server_negotiation.cc:206] Beginning negotiation
0929 17:57:00.946294 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 17:57:00.946336 (+ 42us) server_negotiation.cc:407] Connection header received
0929 17:57:00.946413 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:57:00.946418 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:57:00.946643 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:57:00.946801 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:57:00.947125 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:00.948005 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:00.949643 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:00.950445 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:00.950546 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:00.950719 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0929 17:57:00.951113 (+ 394us) server_negotiation.cc:299] Negotiation successful
0929 17:57:00.951217 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":260,"thread_start_us":77,"threads_started":1}
I20260929 17:57:00.951253 19519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:00.945056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40635 (local address 127.0.0.1:53580)
0929 17:57:00.945525 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0929 17:57:00.945575 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0929 17:57:00.945704 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:57:00.946941 (+ 1237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:57:00.946944 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:57:00.946962 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:57:00.947059 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:57:00.947065 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:57:00.948411 (+ 1346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:57:00.948414 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:57:00.949589 (+ 1175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:57:00.949597 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:57:00.950783 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:57:00.950787 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:57:00.950931 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:00.950968 (+ 37us) client_negotiation.cc:769] Sending connection context
0929 17:57:00.951036 (+ 68us) client_negotiation.cc:240] Negotiation successful
0929 17:57:00.951117 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"mutex_wait_us":69,"thread_start_us":76,"threads_started":1}
I20260929 17:57:00.951782 19521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:00.945218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53580 (local address 127.0.98.3:40635)
0929 17:57:00.946491 (+ 1273us) server_negotiation.cc:206] Beginning negotiation
0929 17:57:00.946501 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 17:57:00.946520 (+ 19us) server_negotiation.cc:407] Connection header received
0929 17:57:00.946588 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:57:00.946593 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:57:00.946801 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:57:00.946946 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:57:00.947187 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:00.948296 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:00.949711 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:00.950445 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:00.950546 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:00.950727 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0929 17:57:00.951526 (+ 799us) server_negotiation.cc:299] Negotiation successful
0929 17:57:00.951622 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1187,"thread_start_us":127,"threads_started":1}
I20260929 17:57:00.951447 19520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:00.945768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35217 (local address 127.0.0.1:51890)
0929 17:57:00.946188 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0929 17:57:00.946199 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 17:57:00.946284 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:57:00.946842 (+ 558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:57:00.946848 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:57:00.946873 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:57:00.947018 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:57:00.947025 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:57:00.948098 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:57:00.948101 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:57:00.949500 (+ 1399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:57:00.949511 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:57:00.950783 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:57:00.950787 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:57:00.950935 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:00.950974 (+ 39us) client_negotiation.cc:769] Sending connection context
0929 17:57:00.951036 (+ 62us) client_negotiation.cc:240] Negotiation successful
0929 17:57:00.951348 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":367,"mutex_wait_us":312,"thread_start_us":108,"threads_started":1}
I20260929 17:57:00.954345 19442 tablet_service.cc:1467] Processing CreateTablet for tablet 4b183ec6067c473baf95a7fee8b47a1c (DEFAULT_TABLE table=test-table [id=a9860d105e4846e6ab35b19f71b5122e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 17:57:00.954428 19310 tablet_service.cc:1467] Processing CreateTablet for tablet 4b183ec6067c473baf95a7fee8b47a1c (DEFAULT_TABLE table=test-table [id=a9860d105e4846e6ab35b19f71b5122e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 17:57:00.954428 19180 tablet_service.cc:1467] Processing CreateTablet for tablet 4b183ec6067c473baf95a7fee8b47a1c (DEFAULT_TABLE table=test-table [id=a9860d105e4846e6ab35b19f71b5122e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 17:57:00.955255 19442 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b183ec6067c473baf95a7fee8b47a1c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:57:00.955392 19180 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b183ec6067c473baf95a7fee8b47a1c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:57:00.955392 19310 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b183ec6067c473baf95a7fee8b47a1c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:57:00.961714 19525 tablet_bootstrap.cc:492] T 4b183ec6067c473baf95a7fee8b47a1c P bd0a0ea4e780444da9109e10a6313db0: Bootstrap starting.
I20260929 17:57:00.962787 19527 tablet_bootstrap.cc:492] T 4b183ec6067c473baf95a7fee8b47a1c P 8c67995864854c3eaf6867744cbc9c0f: Bootstrap starting.
I20260929 17:57:00.963080 19526 tablet_bootstrap.cc:492] T 4b183ec6067c473baf95a7fee8b47a1c P bce413dfb85f4434826050803219ca0a: Bootstrap starting.
I20260929 17:57:00.963864 19525 tablet_bootstrap.cc:654] T 4b183ec6067c473baf95a7fee8b47a1c P bd0a0ea4e780444da9109e10a6313db0: Neither blocks nor log segments found. Creating new log.
I20260929 17:57:00.964561 19525 log.cc:824] T 4b183ec6067c473baf95a7fee8b47a1c P bd0a0ea4e780444da9109e10a6313db0: Log is configured to *not* fsync() on all Append() calls
I20260929 17:57:00.964947 19527 tablet_bootstrap.cc:654] T 4b183ec6067c473baf95a7fee8b47a1c P 8c67995864854c3eaf6867744cbc9c0f: Neither blocks nor log segments found. Creating new log.
I20260929 17:57:00.965063 19526 tablet_bootstrap.cc:654] T 4b183ec6067c473baf95a7fee8b47a1c P bce413dfb85f4434826050803219ca0a: Neither blocks nor log segments found. Creating new log.
I20260929 17:57:00.965796 19526 log.cc:824] T 4b183ec6067c473baf95a7fee8b47a1c P bce413dfb85f4434826050803219ca0a: Log is configured to *not* fsync() on all Append() calls
I20260929 17:57:00.965795 19527 log.cc:824] T 4b183ec6067c473baf95a7fee8b47a1c P 8c67995864854c3eaf6867744cbc9c0f: Log is configured to *not* fsync() on all Append() calls
I20260929 17:57:00.966706 19525 tablet_bootstrap.cc:492] T 4b183ec6067c473baf95a7fee8b47a1c P bd0a0ea4e780444da9109e10a6313db0: No bootstrap required, opened a new log
I20260929 17:57:00.966908 19525 ts_tablet_manager.cc:1397] T 4b183ec6067c473baf95a7fee8b47a1c P bd0a0ea4e780444da9109e10a6313db0: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260929 17:57:00.968438 19526 tablet_bootstrap.cc:492] T 4b183ec6067c473baf95a7fee8b47a1c P bce413dfb85f4434826050803219ca0a: No bootstrap required, opened a new log
I20260929 17:57:00.968452 19527 tablet_bootstrap.cc:492] T 4b183ec6067c473baf95a7fee8b47a1c P 8c67995864854c3eaf6867744cbc9c0f: No bootstrap required, opened a new log
I20260929 17:57:00.968638 19526 ts_tablet_manager.cc:1397] T 4b183ec6067c473baf95a7fee8b47a1c P bce413dfb85f4434826050803219ca0a: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.003s
I20260929 17:57:00.968639 19527 ts_tablet_manager.cc:1397] T 4b183ec6067c473baf95a7fee8b47a1c P 8c67995864854c3eaf6867744cbc9c0f: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260929 17:57:00.971462 19525 raft_consensus.cc:348] T 4b183ec6067c473baf95a7fee8b47a1c P bd0a0ea4e780444da9109e10a6313db0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd0a0ea4e780444da9109e10a6313db0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40635 } } peers { permanent_uuid: "bce413dfb85f4434826050803219ca0a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46303 } } peers { permanent_uuid: "8c67995864854c3eaf6867744cbc9c0f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35217 } }
I20260929 17:57:00.971804 19525 raft_consensus.cc:374] T 4b183ec6067c473baf95a7fee8b47a1c P bd0a0ea4e780444da9109e10a6313db0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:57:00.971872 19525 raft_consensus.cc:729] T 4b183ec6067c473baf95a7fee8b47a1c P bd0a0ea4e780444da9109e10a6313db0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd0a0ea4e780444da9109e10a6313db0, State: Initialized, Role: FOLLOWER
I20260929 17:57:00.971905 19526 raft_consensus.cc:348] T 4b183ec6067c473baf95a7fee8b47a1c P bce413dfb85f4434826050803219ca0a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd0a0ea4e780444da9109e10a6313db0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40635 } } peers { permanent_uuid: "bce413dfb85f4434826050803219ca0a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46303 } } peers { permanent_uuid: "8c67995864854c3eaf6867744cbc9c0f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35217 } }
I20260929 17:57:00.972157 19526 raft_consensus.cc:374] T 4b183ec6067c473baf95a7fee8b47a1c P bce413dfb85f4434826050803219ca0a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:57:00.972219 19526 raft_consensus.cc:729] T 4b183ec6067c473baf95a7fee8b47a1c P bce413dfb85f4434826050803219ca0a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bce413dfb85f4434826050803219ca0a, State: Initialized, Role: FOLLOWER
I20260929 17:57:00.972370 19525 consensus_queue.cc:260] T 4b183ec6067c473baf95a7fee8b47a1c P bd0a0ea4e780444da9109e10a6313db0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd0a0ea4e780444da9109e10a6313db0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40635 } } peers { permanent_uuid: "bce413dfb85f4434826050803219ca0a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46303 } } peers { permanent_uuid: "8c67995864854c3eaf6867744cbc9c0f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35217 } }
I20260929 17:57:00.972651 19526 consensus_queue.cc:260] T 4b183ec6067c473baf95a7fee8b47a1c P bce413dfb85f4434826050803219ca0a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd0a0ea4e780444da9109e10a6313db0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40635 } } peers { permanent_uuid: "bce413dfb85f4434826050803219ca0a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46303 } } peers { permanent_uuid: "8c67995864854c3eaf6867744cbc9c0f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35217 } }
I20260929 17:57:00.972741 19527 raft_consensus.cc:348] T 4b183ec6067c473baf95a7fee8b47a1c P 8c67995864854c3eaf6867744cbc9c0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd0a0ea4e780444da9109e10a6313db0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40635 } } peers { permanent_uuid: "bce413dfb85f4434826050803219ca0a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46303 } } peers { permanent_uuid: "8c67995864854c3eaf6867744cbc9c0f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35217 } }
I20260929 17:57:00.972993 19527 raft_consensus.cc:374] T 4b183ec6067c473baf95a7fee8b47a1c P 8c67995864854c3eaf6867744cbc9c0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:57:00.973078 19527 raft_consensus.cc:729] T 4b183ec6067c473baf95a7fee8b47a1c P 8c67995864854c3eaf6867744cbc9c0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c67995864854c3eaf6867744cbc9c0f, State: Initialized, Role: FOLLOWER
I20260929 17:57:00.973364 19508 heartbeater.cc:503] Master 127.0.98.62:39937 was elected leader, sending a full tablet report...
I20260929 17:57:00.973536 19525 ts_tablet_manager.cc:1428] T 4b183ec6067c473baf95a7fee8b47a1c P bd0a0ea4e780444da9109e10a6313db0: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260929 17:57:00.973519 19527 consensus_queue.cc:260] T 4b183ec6067c473baf95a7fee8b47a1c P 8c67995864854c3eaf6867744cbc9c0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd0a0ea4e780444da9109e10a6313db0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40635 } } peers { permanent_uuid: "bce413dfb85f4434826050803219ca0a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46303 } } peers { permanent_uuid: "8c67995864854c3eaf6867744cbc9c0f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35217 } }
I20260929 17:57:00.973659 19526 ts_tablet_manager.cc:1428] T 4b183ec6067c473baf95a7fee8b47a1c P bce413dfb85f4434826050803219ca0a: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
I20260929 17:57:00.974006 19246 heartbeater.cc:503] Master 127.0.98.62:39937 was elected leader, sending a full tablet report...
I20260929 17:57:00.974123 19377 heartbeater.cc:503] Master 127.0.98.62:39937 was elected leader, sending a full tablet report...
I20260929 17:57:00.975533 19527 ts_tablet_manager.cc:1428] T 4b183ec6067c473baf95a7fee8b47a1c P 8c67995864854c3eaf6867744cbc9c0f: Time spent starting tablet: real 0.007s user 0.003s sys 0.003s
I20260929 17:57:00.977219 19531 raft_consensus.cc:482] T 4b183ec6067c473baf95a7fee8b47a1c P bd0a0ea4e780444da9109e10a6313db0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 17:57:00.977375 19531 raft_consensus.cc:504] T 4b183ec6067c473baf95a7fee8b47a1c P bd0a0ea4e780444da9109e10a6313db0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd0a0ea4e780444da9109e10a6313db0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40635 } } peers { permanent_uuid: "bce413dfb85f4434826050803219ca0a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46303 } } peers { permanent_uuid: "8c67995864854c3eaf6867744cbc9c0f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35217 } }
I20260929 17:57:00.978761 19531 leader_election.cc:288] T 4b183ec6067c473baf95a7fee8b47a1c P bd0a0ea4e780444da9109e10a6313db0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bce413dfb85f4434826050803219ca0a (127.0.98.1:46303), 8c67995864854c3eaf6867744cbc9c0f (127.0.98.2:35217)
I20260929 17:57:00.982707 19510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:00.978943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46303 (local address 127.0.98.3:40329)
0929 17:57:00.979082 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0929 17:57:00.979095 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 17:57:00.979171 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:57:00.979598 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:57:00.979600 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:57:00.979615 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:57:00.979726 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:57:00.979734 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:57:00.980576 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:57:00.980579 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:57:00.981600 (+ 1021us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:57:00.981610 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:57:00.982271 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:57:00.982274 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:57:00.982373 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:00.982399 (+ 26us) client_negotiation.cc:769] Sending connection context
0929 17:57:00.982485 (+ 86us) client_negotiation.cc:240] Negotiation successful
0929 17:57:00.982557 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61,"mutex_wait_us":17}
I20260929 17:57:00.982883 19534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:00.979195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35217 (local address 127.0.98.3:54197)
0929 17:57:00.979598 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0929 17:57:00.979611 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 17:57:00.979695 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:57:00.979900 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:57:00.979903 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:57:00.979918 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:57:00.980029 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:57:00.980035 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:57:00.980831 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:57:00.980834 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:57:00.981804 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:57:00.981814 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:57:00.982533 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:57:00.982536 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:57:00.982610 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:00.982629 (+ 19us) client_negotiation.cc:769] Sending connection context
0929 17:57:00.982683 (+ 54us) client_negotiation.cc:240] Negotiation successful
0929 17:57:00.982735 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":340,"mutex_wait_us":17,"thread_start_us":105,"threads_started":1}
I20260929 17:57:00.982901 19524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:00.979282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54197 (local address 127.0.98.2:35217)
0929 17:57:00.979390 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0929 17:57:00.979394 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:57:00.979679 (+ 285us) server_negotiation.cc:407] Connection header received
0929 17:57:00.979773 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:57:00.979776 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:57:00.979821 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:57:00.979883 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:57:00.980159 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:00.980727 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:00.981909 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:00.982410 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:00.982514 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:00.982565 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0929 17:57:00.982711 (+ 146us) server_negotiation.cc:299] Negotiation successful
0929 17:57:00.982782 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 17:57:00.983052 19523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:00.978991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40329 (local address 127.0.98.1:46303)
0929 17:57:00.979097 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0929 17:57:00.979101 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:57:00.979147 (+ 46us) server_negotiation.cc:407] Connection header received
0929 17:57:00.979455 (+ 308us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:57:00.979459 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:57:00.979506 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:57:00.979578 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:57:00.979831 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:00.980432 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:00.981711 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:00.982175 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:00.982259 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:00.982308 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0929 17:57:00.982862 (+ 554us) server_negotiation.cc:299] Negotiation successful
0929 17:57:00.982947 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260929 17:57:00.983654 19331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b183ec6067c473baf95a7fee8b47a1c" candidate_uuid: "bd0a0ea4e780444da9109e10a6313db0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c67995864854c3eaf6867744cbc9c0f" is_pre_election: true
I20260929 17:57:00.983654 19200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b183ec6067c473baf95a7fee8b47a1c" candidate_uuid: "bd0a0ea4e780444da9109e10a6313db0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bce413dfb85f4434826050803219ca0a" is_pre_election: true
I20260929 17:57:00.983948 19200 raft_consensus.cc:2454] T 4b183ec6067c473baf95a7fee8b47a1c P bce413dfb85f4434826050803219ca0a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bd0a0ea4e780444da9109e10a6313db0 in term 0.
I20260929 17:57:00.983948 19331 raft_consensus.cc:2454] T 4b183ec6067c473baf95a7fee8b47a1c P 8c67995864854c3eaf6867744cbc9c0f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bd0a0ea4e780444da9109e10a6313db0 in term 0.
I20260929 17:57:00.984421 19396 leader_election.cc:302] T 4b183ec6067c473baf95a7fee8b47a1c P bd0a0ea4e780444da9109e10a6313db0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c67995864854c3eaf6867744cbc9c0f, bd0a0ea4e780444da9109e10a6313db0; no voters:
I20260929 17:57:00.984670 19531 raft_consensus.cc:2781] T 4b183ec6067c473baf95a7fee8b47a1c P bd0a0ea4e780444da9109e10a6313db0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 17:57:00.984747 19531 raft_consensus.cc:482] T 4b183ec6067c473baf95a7fee8b47a1c P bd0a0ea4e780444da9109e10a6313db0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 17:57:00.984798 19531 raft_consensus.cc:3037] T 4b183ec6067c473baf95a7fee8b47a1c P bd0a0ea4e780444da9109e10a6313db0 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:57:00.985955 19531 raft_consensus.cc:504] T 4b183ec6067c473baf95a7fee8b47a1c P bd0a0ea4e780444da9109e10a6313db0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd0a0ea4e780444da9109e10a6313db0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40635 } } peers { permanent_uuid: "bce413dfb85f4434826050803219ca0a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46303 } } peers { permanent_uuid: "8c67995864854c3eaf6867744cbc9c0f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35217 } }
I20260929 17:57:00.986285 19531 leader_election.cc:288] T 4b183ec6067c473baf95a7fee8b47a1c P bd0a0ea4e780444da9109e10a6313db0 [CANDIDATE]: Term 1 election: Requested vote from peers bce413dfb85f4434826050803219ca0a (127.0.98.1:46303), 8c67995864854c3eaf6867744cbc9c0f (127.0.98.2:35217)
I20260929 17:57:00.986595 19200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b183ec6067c473baf95a7fee8b47a1c" candidate_uuid: "bd0a0ea4e780444da9109e10a6313db0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bce413dfb85f4434826050803219ca0a"
I20260929 17:57:00.986729 19331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b183ec6067c473baf95a7fee8b47a1c" candidate_uuid: "bd0a0ea4e780444da9109e10a6313db0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c67995864854c3eaf6867744cbc9c0f"
I20260929 17:57:00.986725 19200 raft_consensus.cc:3037] T 4b183ec6067c473baf95a7fee8b47a1c P bce413dfb85f4434826050803219ca0a [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:57:00.986842 19331 raft_consensus.cc:3037] T 4b183ec6067c473baf95a7fee8b47a1c P 8c67995864854c3eaf6867744cbc9c0f [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:57:00.988006 19331 raft_consensus.cc:2454] T 4b183ec6067c473baf95a7fee8b47a1c P 8c67995864854c3eaf6867744cbc9c0f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bd0a0ea4e780444da9109e10a6313db0 in term 1.
I20260929 17:57:00.988000 19200 raft_consensus.cc:2454] T 4b183ec6067c473baf95a7fee8b47a1c P bce413dfb85f4434826050803219ca0a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bd0a0ea4e780444da9109e10a6313db0 in term 1.
I20260929 17:57:00.988299 19395 leader_election.cc:302] T 4b183ec6067c473baf95a7fee8b47a1c P bd0a0ea4e780444da9109e10a6313db0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bce413dfb85f4434826050803219ca0a, bd0a0ea4e780444da9109e10a6313db0; no voters:
I20260929 17:57:00.988457 19531 raft_consensus.cc:2781] T 4b183ec6067c473baf95a7fee8b47a1c P bd0a0ea4e780444da9109e10a6313db0 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 17:57:00.988680 19531 raft_consensus.cc:686] T 4b183ec6067c473baf95a7fee8b47a1c P bd0a0ea4e780444da9109e10a6313db0 [term 1 LEADER]: Becoming Leader. State: Replica: bd0a0ea4e780444da9109e10a6313db0, State: Running, Role: LEADER
I20260929 17:57:00.989056 19531 consensus_queue.cc:237] T 4b183ec6067c473baf95a7fee8b47a1c P bd0a0ea4e780444da9109e10a6313db0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd0a0ea4e780444da9109e10a6313db0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40635 } } peers { permanent_uuid: "bce413dfb85f4434826050803219ca0a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46303 } } peers { permanent_uuid: "8c67995864854c3eaf6867744cbc9c0f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35217 } }
I20260929 17:57:00.992002 19055 catalog_manager.cc:5495] T 4b183ec6067c473baf95a7fee8b47a1c P bd0a0ea4e780444da9109e10a6313db0 reported cstate change: term changed from 0 to 1, leader changed from <none> to bd0a0ea4e780444da9109e10a6313db0 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "bd0a0ea4e780444da9109e10a6313db0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd0a0ea4e780444da9109e10a6313db0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40635 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bce413dfb85f4434826050803219ca0a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46303 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c67995864854c3eaf6867744cbc9c0f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35217 } health_report { overall_health: UNKNOWN } } }
I20260929 17:57:01.004734 19521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:01.001243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53586 (local address 127.0.98.3:40635)
0929 17:57:01.001352 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0929 17:57:01.001355 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 17:57:01.001368 (+ 13us) server_negotiation.cc:407] Connection header received
0929 17:57:01.001440 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:57:01.001442 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:57:01.001496 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:57:01.001588 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 17:57:01.001888 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:01.002408 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:01.003303 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:01.003530 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:01.003620 (+ 90us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:01.003914 (+ 294us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 17:57:01.004327 (+ 413us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 17:57:01.004434 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0929 17:57:01.004524 (+ 90us) server_negotiation.cc:299] Negotiation successful
0929 17:57:01.004597 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260929 17:57:01.011119 19200 raft_consensus.cc:1261] T 4b183ec6067c473baf95a7fee8b47a1c P bce413dfb85f4434826050803219ca0a [term 1 FOLLOWER]: Refusing update from remote peer bd0a0ea4e780444da9109e10a6313db0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 17:57:01.011121 19331 raft_consensus.cc:1261] T 4b183ec6067c473baf95a7fee8b47a1c P 8c67995864854c3eaf6867744cbc9c0f [term 1 FOLLOWER]: Refusing update from remote peer bd0a0ea4e780444da9109e10a6313db0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 17:57:01.011847 19531 consensus_queue.cc:1035] T 4b183ec6067c473baf95a7fee8b47a1c P bd0a0ea4e780444da9109e10a6313db0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c67995864854c3eaf6867744cbc9c0f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 17:57:01.012032 19535 consensus_queue.cc:1035] T 4b183ec6067c473baf95a7fee8b47a1c P bd0a0ea4e780444da9109e10a6313db0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bce413dfb85f4434826050803219ca0a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 17:57:01.018942 19538 mvcc.cc:205] Tried to move back new op lower bound from 7334726127653838848 to 7334726127573602304. Current Snapshot: MvccSnapshot[applied={T|T < 7334726127653838848}]
I20260929 17:57:01.019843 19539 mvcc.cc:205] Tried to move back new op lower bound from 7334726127653838848 to 7334726127573602304. Current Snapshot: MvccSnapshot[applied={T|T < 7334726127653838848}]
I20260929 17:57:01.021260 19540 mvcc.cc:205] Tried to move back new op lower bound from 7334726127653838848 to 7334726127573602304. Current Snapshot: MvccSnapshot[applied={T|T < 7334726127653838848}]
I20260929 17:57:01.030905 19057 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39540:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 17:57:01.031071 19057 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39540:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 17:57:01.033509 19057 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5539e7f336684188a21ea0d971351f58: Sending DeleteTablet for 3 replicas of tablet 4b183ec6067c473baf95a7fee8b47a1c
I20260929 17:57:01.034310 19442 tablet_service.cc:1514] Processing DeleteTablet for tablet 4b183ec6067c473baf95a7fee8b47a1c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 17:57:01 UTC) from {username='slave'} at 127.0.0.1:53580
I20260929 17:57:01.034421 19310 tablet_service.cc:1514] Processing DeleteTablet for tablet 4b183ec6067c473baf95a7fee8b47a1c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 17:57:01 UTC) from {username='slave'} at 127.0.0.1:51890
I20260929 17:57:01.034538 19180 tablet_service.cc:1514] Processing DeleteTablet for tablet 4b183ec6067c473baf95a7fee8b47a1c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 17:57:01 UTC) from {username='slave'} at 127.0.0.1:35952
I20260929 17:57:01.035120 19550 tablet_replica.cc:330] T 4b183ec6067c473baf95a7fee8b47a1c P bce413dfb85f4434826050803219ca0a: stopping tablet replica
I20260929 17:57:01.035449 19550 raft_consensus.cc:2229] T 4b183ec6067c473baf95a7fee8b47a1c P bce413dfb85f4434826050803219ca0a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 17:57:01.035493 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskup3Soe/build/debug/bin/kudu with pid 19118
I20260929 17:57:01.035801 19550 raft_consensus.cc:2258] T 4b183ec6067c473baf95a7fee8b47a1c P bce413dfb85f4434826050803219ca0a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 17:57:01.036101 19548 tablet_replica.cc:330] T 4b183ec6067c473baf95a7fee8b47a1c P bd0a0ea4e780444da9109e10a6313db0: stopping tablet replica
I20260929 17:57:01.036851 19548 raft_consensus.cc:2229] T 4b183ec6067c473baf95a7fee8b47a1c P bd0a0ea4e780444da9109e10a6313db0 [term 1 LEADER]: Raft consensus shutting down.
I20260929 17:57:01.037191 19549 tablet_replica.cc:330] T 4b183ec6067c473baf95a7fee8b47a1c P 8c67995864854c3eaf6867744cbc9c0f: stopping tablet replica
I20260929 17:57:01.037278 19548 raft_consensus.cc:2258] T 4b183ec6067c473baf95a7fee8b47a1c P bd0a0ea4e780444da9109e10a6313db0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 17:57:01.037686 19549 raft_consensus.cc:2229] T 4b183ec6067c473baf95a7fee8b47a1c P 8c67995864854c3eaf6867744cbc9c0f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 17:57:01.038026 19549 raft_consensus.cc:2258] T 4b183ec6067c473baf95a7fee8b47a1c P 8c67995864854c3eaf6867744cbc9c0f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 17:57:01.038812 19548 ts_tablet_manager.cc:1905] T 4b183ec6067c473baf95a7fee8b47a1c P bd0a0ea4e780444da9109e10a6313db0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 17:57:01.039572 19549 ts_tablet_manager.cc:1905] T 4b183ec6067c473baf95a7fee8b47a1c P 8c67995864854c3eaf6867744cbc9c0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 17:57:01.041481 19548 ts_tablet_manager.cc:1918] T 4b183ec6067c473baf95a7fee8b47a1c P bd0a0ea4e780444da9109e10a6313db0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 17:57:01.041574 19548 log.cc:1196] T 4b183ec6067c473baf95a7fee8b47a1c P bd0a0ea4e780444da9109e10a6313db0: Deleting WAL directory at /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790704375975685-392-0/minicluster-data/ts-2/wal/wals/4b183ec6067c473baf95a7fee8b47a1c
I20260929 17:57:01.042312 19549 ts_tablet_manager.cc:1918] T 4b183ec6067c473baf95a7fee8b47a1c P 8c67995864854c3eaf6867744cbc9c0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 17:57:01.042362 19548 ts_tablet_manager.cc:1939] T 4b183ec6067c473baf95a7fee8b47a1c P bd0a0ea4e780444da9109e10a6313db0: Deleting consensus metadata
I20260929 17:57:01.042512 19549 log.cc:1196] T 4b183ec6067c473baf95a7fee8b47a1c P 8c67995864854c3eaf6867744cbc9c0f: Deleting WAL directory at /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790704375975685-392-0/minicluster-data/ts-1/wal/wals/4b183ec6067c473baf95a7fee8b47a1c
I20260929 17:57:01.042855 19549 ts_tablet_manager.cc:1939] T 4b183ec6067c473baf95a7fee8b47a1c P 8c67995864854c3eaf6867744cbc9c0f: Deleting consensus metadata
I20260929 17:57:01.043726 19043 catalog_manager.cc:4841] TS bd0a0ea4e780444da9109e10a6313db0 (127.0.98.3:40635): tablet 4b183ec6067c473baf95a7fee8b47a1c (table test-table [id=a9860d105e4846e6ab35b19f71b5122e]) successfully deleted
I20260929 17:57:01.044205 19043 catalog_manager.cc:4841] TS 8c67995864854c3eaf6867744cbc9c0f (127.0.98.2:35217): tablet 4b183ec6067c473baf95a7fee8b47a1c (table test-table [id=a9860d105e4846e6ab35b19f71b5122e]) successfully deleted
W20260929 17:57:01.044539 19042 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:46303 (error 108)
I20260929 17:57:01.044760 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskup3Soe/build/debug/bin/kudu with pid 19249
I20260929 17:57:01.046793 19520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:01.046496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46303 (local address 127.0.0.1:35954)
0929 17:57:01.046631 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0929 17:57:01.046701 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260929 17:57:01.046972 19042 catalog_manager.cc:4584] TS bce413dfb85f4434826050803219ca0a (127.0.98.1:46303): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4b183ec6067c473baf95a7fee8b47a1c: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46303: connect: Connection refused (error 111)
I20260929 17:57:01.050028 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskup3Soe/build/debug/bin/kudu with pid 19380
I20260929 17:57:01.054733 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskup3Soe/build/debug/bin/kudu with pid 19027
2026-09-29T17:57:01Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (991 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704375975685-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 17:57:01 dist-test-slave-53nh krb5kdc[19554](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 17:57:01 dist-test-slave-53nh krb5kdc[19554](info): set up 2 sockets
Sep 29 17:57:01 dist-test-slave-53nh krb5kdc[19554](info): commencing operation
krb5kdc: starting...
W20260929 17:57:02.005053 392 mini_kdc.cc:121] Time spent starting KDC: real 0.931s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704375975685-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704375975685-392-0/krb5kdc/test-user.keytab.
Sep 29 17:57:02 dist-test-slave-53nh krb5kdc[19554](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704622, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T17:57:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T17:57:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704375975685-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704375975685-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 17:57:02.124414 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskup3Soe/build/debug/bin/kudu
/tmp/dist-test-taskup3Soe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704375975685-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704375975685-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704375975685-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704375975685-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40557
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704375975685-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40557
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704375975685-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704375975685-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704375975685-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:57:02.233731 19567 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:57:02.234025 19567 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:57:02.234081 19567 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:57:02.238943 19567 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 17:57:02.239015 19567 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:57:02.239041 19567 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:57:02.239064 19567 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 17:57:02.239094 19567 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 17:57:02.244652 19567 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704375975685-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704375975685-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40557
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704375975685-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40557
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704375975685-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19567
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704375975685-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:57:02.245833 19567 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:57:02.246896 19567 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:57:02.255574 19572 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:57:02.255574 19575 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:57:02.255577 19573 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:57:02.255673 19567 server_base.cc:1034] running on GCE node
I20260929 17:57:02.256403 19567 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:57:02.257573 19567 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:57:02.258733 19567 hybrid_clock.cc:648] HybridClock initialized: now 1790704622258717 us; error 29 us; skew 500 ppm
Sep 29 17:57:02 dist-test-slave-53nh krb5kdc[19554](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704622, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:57:02.261634 19567 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 17:57:02.262573 19567 webserver.cc:460] Webserver started at http://127.0.98.62:36299/ using document root <none> and password file <none>
I20260929 17:57:02.263111 19567 fs_manager.cc:359] Metadata directory not provided
I20260929 17:57:02.263167 19567 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:57:02.263343 19567 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:57:02.265400 19567 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704375975685-392-0/minicluster-data/master-0/data/instance:
uuid: "9fbe87dbf95e44bea93bac28f7df4e06"
format_stamp: "Formatted at 2026-09-29 17:57:02 on dist-test-slave-53nh"
server_key: "99dff3e0d1b69f51685a3237c8406cb8"
server_key_iv: "e1e509391d5fc9a350dc87770f615f95"
server_key_version: "encryptionkey@0"
I20260929 17:57:02.265900 19567 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704375975685-392-0/minicluster-data/master-0/wal/instance:
uuid: "9fbe87dbf95e44bea93bac28f7df4e06"
format_stamp: "Formatted at 2026-09-29 17:57:02 on dist-test-slave-53nh"
server_key: "99dff3e0d1b69f51685a3237c8406cb8"
server_key_iv: "e1e509391d5fc9a350dc87770f615f95"
server_key_version: "encryptionkey@0"
I20260929 17:57:02.269575 19567 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260929 17:57:02.272058 19582 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:57:02.273456 19567 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 17:57:02.273576 19567 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704375975685-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704375975685-392-0/minicluster-data/master-0/wal
uuid: "9fbe87dbf95e44bea93bac28f7df4e06"
format_stamp: "Formatted at 2026-09-29 17:57:02 on dist-test-slave-53nh"
server_key: "99dff3e0d1b69f51685a3237c8406cb8"
server_key_iv: "e1e509391d5fc9a350dc87770f615f95"
server_key_version: "encryptionkey@0"
I20260929 17:57:02.273674 19567 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704375975685-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704375975685-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704375975685-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:57:02.288741 19567 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:57:02.292212 19567 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:57:02.292369 19567 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:57:02.299142 19567 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40557
I20260929 17:57:02.299183 19634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40557 every 8 connection(s)
I20260929 17:57:02.300063 19567 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704375975685-392-0/minicluster-data/master-0/data/info.pb
I20260929 17:57:02.300632 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskup3Soe/build/debug/bin/kudu as pid 19567
I20260929 17:57:02.300758 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704375975685-392-0/minicluster-data/master-0/wal/instance
I20260929 17:57:02.301036 392 external_mini_cluster.cc:1428] Setting key b3f5d9cafb9cb57b4270181de26a4692
I20260929 17:57:02.303164 19635 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:57:02.309617 19635 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9fbe87dbf95e44bea93bac28f7df4e06: Bootstrap starting.
Sep 29 17:57:02 dist-test-slave-53nh krb5kdc[19554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704622, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 17:57:02.312281 19635 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9fbe87dbf95e44bea93bac28f7df4e06: Neither blocks nor log segments found. Creating new log.
I20260929 17:57:02.313120 19635 log.cc:824] T 00000000000000000000000000000000 P 9fbe87dbf95e44bea93bac28f7df4e06: Log is configured to *not* fsync() on all Append() calls
I20260929 17:57:02.314965 19635 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9fbe87dbf95e44bea93bac28f7df4e06: No bootstrap required, opened a new log
I20260929 17:57:02.315603 19638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:02.302404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56702 (local address 127.0.98.62:40557)
0929 17:57:02.302863 (+ 459us) server_negotiation.cc:206] Beginning negotiation
0929 17:57:02.302882 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0929 17:57:02.302925 (+ 43us) server_negotiation.cc:407] Connection header received
0929 17:57:02.303736 (+ 811us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:57:02.303759 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:57:02.304156 (+ 397us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:57:02.304505 (+ 349us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:57:02.305131 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:02.306627 (+ 1496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:02.307462 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:02.308130 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:02.308286 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:02.310889 (+ 2603us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:57:02.310896 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:57:02.310913 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:57:02.310946 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:57:02.312713 (+ 1767us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:57:02.313130 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:57:02.313136 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:57:02.313142 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:57:02.313271 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:57:02.313549 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:57:02.313552 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:57:02.313553 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:57:02.313953 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:57:02.314133 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0929 17:57:02.314403 (+ 270us) server_negotiation.cc:299] Negotiation successful
0929 17:57:02.314679 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":321,"thread_start_us":114,"threads_started":1}
I20260929 17:57:02.317924 19635 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9fbe87dbf95e44bea93bac28f7df4e06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fbe87dbf95e44bea93bac28f7df4e06" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40557 } }
I20260929 17:57:02.318156 19635 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9fbe87dbf95e44bea93bac28f7df4e06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:57:02.318217 19635 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9fbe87dbf95e44bea93bac28f7df4e06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9fbe87dbf95e44bea93bac28f7df4e06, State: Initialized, Role: FOLLOWER
I20260929 17:57:02.318600 19635 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9fbe87dbf95e44bea93bac28f7df4e06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fbe87dbf95e44bea93bac28f7df4e06" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40557 } }
I20260929 17:57:02.318722 19635 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9fbe87dbf95e44bea93bac28f7df4e06 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 17:57:02.318782 19635 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9fbe87dbf95e44bea93bac28f7df4e06 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 17:57:02.318873 19635 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9fbe87dbf95e44bea93bac28f7df4e06 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:57:02.319723 19635 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9fbe87dbf95e44bea93bac28f7df4e06 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fbe87dbf95e44bea93bac28f7df4e06" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40557 } }
I20260929 17:57:02.320025 19635 leader_election.cc:302] T 00000000000000000000000000000000 P 9fbe87dbf95e44bea93bac28f7df4e06 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9fbe87dbf95e44bea93bac28f7df4e06; no voters:
I20260929 17:57:02.320302 19635 leader_election.cc:288] T 00000000000000000000000000000000 P 9fbe87dbf95e44bea93bac28f7df4e06 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 17:57:02.320403 19640 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9fbe87dbf95e44bea93bac28f7df4e06 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 17:57:02.320639 19640 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9fbe87dbf95e44bea93bac28f7df4e06 [term 1 LEADER]: Becoming Leader. State: Replica: 9fbe87dbf95e44bea93bac28f7df4e06, State: Running, Role: LEADER
I20260929 17:57:02.321017 19640 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9fbe87dbf95e44bea93bac28f7df4e06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fbe87dbf95e44bea93bac28f7df4e06" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40557 } }
I20260929 17:57:02.321480 19635 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9fbe87dbf95e44bea93bac28f7df4e06 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 17:57:02.322530 19642 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9fbe87dbf95e44bea93bac28f7df4e06 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9fbe87dbf95e44bea93bac28f7df4e06. Latest consensus state: current_term: 1 leader_uuid: "9fbe87dbf95e44bea93bac28f7df4e06" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fbe87dbf95e44bea93bac28f7df4e06" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40557 } } }
I20260929 17:57:02.322927 19642 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9fbe87dbf95e44bea93bac28f7df4e06 [sys.catalog]: This master's current role is: LEADER
I20260929 17:57:02.322818 19641 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9fbe87dbf95e44bea93bac28f7df4e06 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9fbe87dbf95e44bea93bac28f7df4e06" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fbe87dbf95e44bea93bac28f7df4e06" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40557 } } }
I20260929 17:57:02.323071 19641 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9fbe87dbf95e44bea93bac28f7df4e06 [sys.catalog]: This master's current role is: LEADER
I20260929 17:57:02.323381 19654 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 17:57:02.326290 19654 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 17:57:02.332571 19654 catalog_manager.cc:1347] Generated new cluster ID: ea4d8cc476ae4a1f848599d876b9d2e6
I20260929 17:57:02.332669 19654 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 17:57:02.352226 19654 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 17:57:02.353245 19654 catalog_manager.cc:1504] Loading token signing keys...
I20260929 17:57:02.361443 19654 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9fbe87dbf95e44bea93bac28f7df4e06: Generated new TSK 0
I20260929 17:57:02.362177 19654 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704375975685-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704375975685-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260929 17:57:02.398429 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskup3Soe/build/debug/bin/kudu
/tmp/dist-test-taskup3Soe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704375975685-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704375975685-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704375975685-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704375975685-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40557
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704375975685-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704375975685-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704375975685-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704375975685-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:57:02.511904 19661 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:57:02.512190 19661 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:57:02.512235 19661 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:57:02.516387 19661 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:57:02.516463 19661 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:57:02.516527 19661 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 17:57:02.522063 19661 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704375975685-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704375975685-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704375975685-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704375975685-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40557
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19661
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704375975685-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:57:02.523187 19661 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:57:02.524258 19661 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:57:02.533962 19666 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:57:02.533960 19667 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:57:02.534147 19661 server_base.cc:1034] running on GCE node
W20260929 17:57:02.534031 19669 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:57:02.534723 19661 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:57:02.535377 19661 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:57:02.536548 19661 hybrid_clock.cc:648] HybridClock initialized: now 1790704622536519 us; error 43 us; skew 500 ppm
Sep 29 17:57:02 dist-test-slave-53nh krb5kdc[19554](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704622, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:57:02.539310 19661 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260929 17:57:02.540169 19661 webserver.cc:460] Webserver started at http://127.0.98.1:43021/ using document root <none> and password file <none>
I20260929 17:57:02.540722 19661 fs_manager.cc:359] Metadata directory not provided
I20260929 17:57:02.540784 19661 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:57:02.540984 19661 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:57:02.543087 19661 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704375975685-392-0/minicluster-data/ts-0/data/instance:
uuid: "4e6e3f80fc904b92854863c35b3fe1aa"
format_stamp: "Formatted at 2026-09-29 17:57:02 on dist-test-slave-53nh"
server_key: "8b3f6e13a2a250b7b58ff78d7ba1b1fc"
server_key_iv: "a0fa862c08f43fcafc7f431ed49839c5"
server_key_version: "encryptionkey@0"
I20260929 17:57:02.543661 19661 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704375975685-392-0/minicluster-data/ts-0/wal/instance:
uuid: "4e6e3f80fc904b92854863c35b3fe1aa"
format_stamp: "Formatted at 2026-09-29 17:57:02 on dist-test-slave-53nh"
server_key: "8b3f6e13a2a250b7b58ff78d7ba1b1fc"
server_key_iv: "a0fa862c08f43fcafc7f431ed49839c5"
server_key_version: "encryptionkey@0"
I20260929 17:57:02.547634 19661 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260929 17:57:02.550160 19676 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:57:02.551416 19661 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 17:57:02.551525 19661 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704375975685-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704375975685-392-0/minicluster-data/ts-0/wal
uuid: "4e6e3f80fc904b92854863c35b3fe1aa"
format_stamp: "Formatted at 2026-09-29 17:57:02 on dist-test-slave-53nh"
server_key: "8b3f6e13a2a250b7b58ff78d7ba1b1fc"
server_key_iv: "a0fa862c08f43fcafc7f431ed49839c5"
server_key_version: "encryptionkey@0"
I20260929 17:57:02.551645 19661 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704375975685-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704375975685-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704375975685-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:57:02.573424 19661 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:57:02.576647 19661 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:57:02.576874 19661 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:57:02.577476 19661 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 17:57:02.578329 19661 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:57:02.578373 19661 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:57:02.578411 19661 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:57:02.578430 19661 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 17:57:02.587203 19661 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35791
I20260929 17:57:02.587260 19789 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35791 every 8 connection(s)
I20260929 17:57:02.588114 19661 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704375975685-392-0/minicluster-data/ts-0/data/info.pb
I20260929 17:57:02.594924 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskup3Soe/build/debug/bin/kudu as pid 19661
I20260929 17:57:02.595034 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704375975685-392-0/minicluster-data/ts-0/wal/instance
I20260929 17:57:02.595306 392 external_mini_cluster.cc:1428] Setting key a115443988887a9d9fa5dda7518b9bd6
Sep 29 17:57:02 dist-test-slave-53nh krb5kdc[19554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704622, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 17:57:02.602805 19638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:02.590058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40933 (local address 127.0.98.62:40557)
0929 17:57:02.590192 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0929 17:57:02.590195 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 17:57:02.591008 (+ 813us) server_negotiation.cc:407] Connection header received
0929 17:57:02.592301 (+ 1293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:57:02.592305 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:57:02.592372 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:57:02.592457 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:57:02.593790 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:02.594337 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:02.595626 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:02.595903 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:02.596018 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:02.598825 (+ 2807us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:57:02.598829 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:57:02.598832 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:57:02.598862 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:57:02.600280 (+ 1418us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:57:02.601030 (+ 750us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:57:02.601034 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:57:02.601036 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:57:02.601140 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:57:02.601553 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:57:02.601556 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:57:02.601557 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:57:02.601806 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:57:02.601894 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0929 17:57:02.602530 (+ 636us) server_negotiation.cc:299] Negotiation successful
0929 17:57:02.602645 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 17:57:02.603995 19792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:02.590377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40557 (local address 127.0.98.1:40933)
0929 17:57:02.590850 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0929 17:57:02.590895 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 17:57:02.592064 (+ 1169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:57:02.592695 (+ 631us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:57:02.592702 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:57:02.593114 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:57:02.593625 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:57:02.593640 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:57:02.594471 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:57:02.594473 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:57:02.595500 (+ 1027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:57:02.595511 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:57:02.596075 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:57:02.596079 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:57:02.596327 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:02.596969 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:57:02.597022 (+ 53us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:57:02.598642 (+ 1620us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:57:02.600394 (+ 1752us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:57:02.600400 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:57:02.600418 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:57:02.600803 (+ 385us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:57:02.601240 (+ 437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:57:02.601243 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:57:02.601245 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:57:02.601438 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:57:02.601910 (+ 472us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:57:02.601916 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:57:02.602269 (+ 353us) client_negotiation.cc:769] Sending connection context
0929 17:57:02.602496 (+ 227us) client_negotiation.cc:240] Negotiation successful
0929 17:57:02.602885 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":101,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260929 17:57:02.605254 19790 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40557
I20260929 17:57:02.605491 19790 heartbeater.cc:464] Registering TS with master...
I20260929 17:57:02.606015 19790 heartbeater.cc:511] Master 127.0.98.62:40557 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260929 17:57:02.607798 19599 ts_manager.cc:194] Registered new tserver with Master: 4e6e3f80fc904b92854863c35b3fe1aa (127.0.98.1:35791)
I20260929 17:57:02.609215 19599 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:40933
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704375975685-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704375975685-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260929 17:57:02.624228 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskup3Soe/build/debug/bin/kudu
/tmp/dist-test-taskup3Soe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704375975685-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704375975685-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704375975685-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704375975685-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40557
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704375975685-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704375975685-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704375975685-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704375975685-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:57:02.739953 19795 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:57:02.740244 19795 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:57:02.740296 19795 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:57:02.744444 19795 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:57:02.744525 19795 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:57:02.744587 19795 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 17:57:02.749959 19795 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704375975685-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704375975685-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704375975685-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704375975685-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40557
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19795
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704375975685-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:57:02.751034 19795 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:57:02.752024 19795 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:57:02.761477 19801 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:57:02.761521 19800 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:57:02.761555 19795 server_base.cc:1034] running on GCE node
W20260929 17:57:02.761659 19803 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:57:02.762094 19795 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:57:02.762746 19795 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:57:02.763912 19795 hybrid_clock.cc:648] HybridClock initialized: now 1790704622763891 us; error 31 us; skew 500 ppm
Sep 29 17:57:02 dist-test-slave-53nh krb5kdc[19554](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704622, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:57:02.766912 19795 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260929 17:57:02.767840 19795 webserver.cc:460] Webserver started at http://127.0.98.2:46821/ using document root <none> and password file <none>
I20260929 17:57:02.768445 19795 fs_manager.cc:359] Metadata directory not provided
I20260929 17:57:02.768525 19795 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:57:02.768761 19795 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:57:02.770957 19795 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704375975685-392-0/minicluster-data/ts-1/data/instance:
uuid: "85d41a99a01847c092aef2c87ee7affc"
format_stamp: "Formatted at 2026-09-29 17:57:02 on dist-test-slave-53nh"
server_key: "523579452b73328af65d6cc3b7ef33de"
server_key_iv: "69842447ac653a964ae4bb7bf201c502"
server_key_version: "encryptionkey@0"
I20260929 17:57:02.771531 19795 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704375975685-392-0/minicluster-data/ts-1/wal/instance:
uuid: "85d41a99a01847c092aef2c87ee7affc"
format_stamp: "Formatted at 2026-09-29 17:57:02 on dist-test-slave-53nh"
server_key: "523579452b73328af65d6cc3b7ef33de"
server_key_iv: "69842447ac653a964ae4bb7bf201c502"
server_key_version: "encryptionkey@0"
I20260929 17:57:02.775540 19795 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 17:57:02.778438 19810 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:57:02.779820 19795 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 17:57:02.779946 19795 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704375975685-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704375975685-392-0/minicluster-data/ts-1/wal
uuid: "85d41a99a01847c092aef2c87ee7affc"
format_stamp: "Formatted at 2026-09-29 17:57:02 on dist-test-slave-53nh"
server_key: "523579452b73328af65d6cc3b7ef33de"
server_key_iv: "69842447ac653a964ae4bb7bf201c502"
server_key_version: "encryptionkey@0"
I20260929 17:57:02.780054 19795 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704375975685-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704375975685-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704375975685-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:57:02.796355 19795 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:57:02.799885 19795 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:57:02.800088 19795 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:57:02.800714 19795 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 17:57:02.801649 19795 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:57:02.801708 19795 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:57:02.801766 19795 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:57:02.801791 19795 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 17:57:02.811067 19795 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38211
I20260929 17:57:02.811087 19923 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38211 every 8 connection(s)
I20260929 17:57:02.812074 19795 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704375975685-392-0/minicluster-data/ts-1/data/info.pb
I20260929 17:57:02.821609 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskup3Soe/build/debug/bin/kudu as pid 19795
I20260929 17:57:02.821722 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704375975685-392-0/minicluster-data/ts-1/wal/instance
I20260929 17:57:02.821986 392 external_mini_cluster.cc:1428] Setting key 781f536f015918a0dc7746e99dc519f4
Sep 29 17:57:02 dist-test-slave-53nh krb5kdc[19554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704622, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 17:57:02.827174 19638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:02.814064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56187 (local address 127.0.98.62:40557)
0929 17:57:02.814212 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0929 17:57:02.814217 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:57:02.815271 (+ 1054us) server_negotiation.cc:407] Connection header received
0929 17:57:02.816595 (+ 1324us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:57:02.816598 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:57:02.816647 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:57:02.816725 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:57:02.818137 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:02.818731 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:02.820225 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:02.820465 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:02.820555 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:02.823687 (+ 3132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:57:02.823691 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:57:02.823694 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:57:02.823723 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:57:02.824970 (+ 1247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:57:02.825616 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:57:02.825619 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:57:02.825620 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:57:02.825705 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:57:02.826067 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:57:02.826069 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:57:02.826070 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:57:02.826304 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:57:02.826406 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0929 17:57:02.826889 (+ 483us) server_negotiation.cc:299] Negotiation successful
0929 17:57:02.827014 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 17:57:02.827991 19926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:02.814511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40557 (local address 127.0.98.2:56187)
0929 17:57:02.815088 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0929 17:57:02.815148 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0929 17:57:02.816368 (+ 1220us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:57:02.816949 (+ 581us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:57:02.816957 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:57:02.817409 (+ 452us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:57:02.817944 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:57:02.817959 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:57:02.818903 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:57:02.818906 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:57:02.820060 (+ 1154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:57:02.820073 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:57:02.820623 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:57:02.820626 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:57:02.820829 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:02.821609 (+ 780us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:57:02.821627 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:57:02.823512 (+ 1885us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:57:02.825129 (+ 1617us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:57:02.825136 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:57:02.825155 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:57:02.825507 (+ 352us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:57:02.825792 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:57:02.825795 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:57:02.825797 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:57:02.825981 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:57:02.826409 (+ 428us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:57:02.826415 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:57:02.826661 (+ 246us) client_negotiation.cc:769] Sending connection context
0929 17:57:02.826822 (+ 161us) client_negotiation.cc:240] Negotiation successful
0929 17:57:02.827090 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":120,"threads_started":1}
I20260929 17:57:02.829396 19924 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40557
I20260929 17:57:02.829635 19924 heartbeater.cc:464] Registering TS with master...
I20260929 17:57:02.830157 19924 heartbeater.cc:511] Master 127.0.98.62:40557 requested a full tablet report, sending...
I20260929 17:57:02.831270 19599 ts_manager.cc:194] Registered new tserver with Master: 85d41a99a01847c092aef2c87ee7affc (127.0.98.2:38211)
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260929 17:57:02.832005 19599 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:56187
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704375975685-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704375975685-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260929 17:57:02.852157 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskup3Soe/build/debug/bin/kudu
/tmp/dist-test-taskup3Soe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704375975685-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704375975685-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704375975685-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704375975685-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40557
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704375975685-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704375975685-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704375975685-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704375975685-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:57:02.963325 19929 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:57:02.963608 19929 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:57:02.963653 19929 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:57:02.967868 19929 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:57:02.967949 19929 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:57:02.968016 19929 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 17:57:02.975466 19929 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704375975685-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704375975685-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704375975685-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704375975685-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40557
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19929
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704375975685-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:57:02.977042 19929 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:57:02.978329 19929 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:57:02.990710 19934 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:57:02.990710 19937 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:57:02.991004 19929 server_base.cc:1034] running on GCE node
W20260929 17:57:02.990705 19935 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:57:02.991604 19929 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:57:02.992470 19929 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:57:02.993661 19929 hybrid_clock.cc:648] HybridClock initialized: now 1790704622993623 us; error 56 us; skew 500 ppm
Sep 29 17:57:02 dist-test-slave-53nh krb5kdc[19554](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704622, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:57:02.997161 19929 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260929 17:57:02.998260 19929 webserver.cc:460] Webserver started at http://127.0.98.3:43197/ using document root <none> and password file <none>
I20260929 17:57:02.998991 19929 fs_manager.cc:359] Metadata directory not provided
I20260929 17:57:02.999078 19929 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:57:02.999289 19929 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:57:03.002068 19929 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704375975685-392-0/minicluster-data/ts-2/data/instance:
uuid: "0af7d6cca2d7458cbfeca8952d007dab"
format_stamp: "Formatted at 2026-09-29 17:57:02 on dist-test-slave-53nh"
server_key: "1f539c252770320cc80703594c017135"
server_key_iv: "7362f082263643dafc8451ef46f670ef"
server_key_version: "encryptionkey@0"
I20260929 17:57:03.002806 19929 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704375975685-392-0/minicluster-data/ts-2/wal/instance:
uuid: "0af7d6cca2d7458cbfeca8952d007dab"
format_stamp: "Formatted at 2026-09-29 17:57:02 on dist-test-slave-53nh"
server_key: "1f539c252770320cc80703594c017135"
server_key_iv: "7362f082263643dafc8451ef46f670ef"
server_key_version: "encryptionkey@0"
I20260929 17:57:03.008241 19929 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 17:57:03.011899 19944 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:57:03.013688 19929 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 17:57:03.013846 19929 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704375975685-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704375975685-392-0/minicluster-data/ts-2/wal
uuid: "0af7d6cca2d7458cbfeca8952d007dab"
format_stamp: "Formatted at 2026-09-29 17:57:02 on dist-test-slave-53nh"
server_key: "1f539c252770320cc80703594c017135"
server_key_iv: "7362f082263643dafc8451ef46f670ef"
server_key_version: "encryptionkey@0"
I20260929 17:57:03.013999 19929 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704375975685-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704375975685-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704375975685-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:57:03.034592 19929 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:57:03.037938 19929 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:57:03.038093 19929 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:57:03.038647 19929 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 17:57:03.039458 19929 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:57:03.039506 19929 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:57:03.039546 19929 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:57:03.039566 19929 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 17:57:03.048420 19929 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37641
I20260929 17:57:03.048544 20057 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37641 every 8 connection(s)
I20260929 17:57:03.049427 19929 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704375975685-392-0/minicluster-data/ts-2/data/info.pb
I20260929 17:57:03.059046 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskup3Soe/build/debug/bin/kudu as pid 19929
I20260929 17:57:03.059190 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704375975685-392-0/minicluster-data/ts-2/wal/instance
Sep 29 17:57:03 dist-test-slave-53nh krb5kdc[19554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704622, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 17:57:03.059504 392 external_mini_cluster.cc:1428] Setting key 3579b60f0d5a1826e22d2973662b5b1f
I20260929 17:57:03.063584 19638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:03.051374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58235 (local address 127.0.98.62:40557)
0929 17:57:03.051508 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0929 17:57:03.051512 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:57:03.052414 (+ 902us) server_negotiation.cc:407] Connection header received
0929 17:57:03.053784 (+ 1370us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:57:03.053788 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:57:03.053852 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:57:03.053929 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:57:03.055234 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:03.055842 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:03.057062 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:03.057286 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:03.057380 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:03.059995 (+ 2615us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:57:03.059999 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:57:03.060002 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:57:03.060035 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:57:03.061668 (+ 1633us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:57:03.062209 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:57:03.062212 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:57:03.062213 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:57:03.062295 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:57:03.062571 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:57:03.062574 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:57:03.062575 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:57:03.062754 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:57:03.062837 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0929 17:57:03.063341 (+ 504us) server_negotiation.cc:299] Negotiation successful
0929 17:57:03.063458 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 17:57:03.064452 20060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:03.051731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40557 (local address 127.0.98.3:58235)
0929 17:57:03.052253 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0929 17:57:03.052298 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 17:57:03.053556 (+ 1258us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:57:03.054133 (+ 577us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:57:03.054142 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:57:03.054563 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:57:03.055074 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:57:03.055088 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:57:03.055979 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:57:03.055982 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:57:03.056922 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:57:03.056932 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:57:03.057437 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:57:03.057441 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:57:03.057623 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:03.058195 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:57:03.058212 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:57:03.059803 (+ 1591us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:57:03.061785 (+ 1982us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:57:03.061791 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:57:03.061808 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:57:03.062106 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:57:03.062361 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:57:03.062364 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:57:03.062365 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:57:03.062495 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:57:03.062835 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:57:03.062841 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:57:03.063078 (+ 237us) client_negotiation.cc:769] Sending connection context
0929 17:57:03.063256 (+ 178us) client_negotiation.cc:240] Negotiation successful
0929 17:57:03.063515 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":114,"threads_started":1}
I20260929 17:57:03.065551 20058 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40557
I20260929 17:57:03.065805 20058 heartbeater.cc:464] Registering TS with master...
I20260929 17:57:03.066303 20058 heartbeater.cc:511] Master 127.0.98.62:40557 requested a full tablet report, sending...
I20260929 17:57:03.067417 19599 ts_manager.cc:194] Registered new tserver with Master: 0af7d6cca2d7458cbfeca8952d007dab (127.0.98.3:37641)
I20260929 17:57:03.067999 19599 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:58235
I20260929 17:57:03.073662 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 17:57:03.082592 19638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:03.075585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56716 (local address 127.0.98.62:40557)
0929 17:57:03.075741 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0929 17:57:03.075746 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:57:03.075895 (+ 149us) server_negotiation.cc:407] Connection header received
0929 17:57:03.075965 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:57:03.075967 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:57:03.076008 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:57:03.076076 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:57:03.076732 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:03.077399 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:03.078332 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:03.078540 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:03.078613 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:03.079956 (+ 1343us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:57:03.079959 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:57:03.079961 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:57:03.079985 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:57:03.081275 (+ 1290us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:57:03.081654 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:57:03.081657 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:57:03.081658 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:57:03.081724 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:57:03.081929 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:57:03.081931 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:57:03.081932 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:57:03.082136 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:57:03.082228 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0929 17:57:03.082378 (+ 150us) server_negotiation.cc:299] Negotiation successful
0929 17:57:03.082468 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260929 17:57:03.085414 19599 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56716:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 17:57:03.088533 19599 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 17:57:03.103291 20073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:03.098691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47560 (local address 127.0.98.3:37641)
0929 17:57:03.099041 (+ 350us) server_negotiation.cc:206] Beginning negotiation
0929 17:57:03.099052 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 17:57:03.099073 (+ 21us) server_negotiation.cc:407] Connection header received
0929 17:57:03.099140 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:57:03.099145 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:57:03.099334 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:57:03.099477 (+ 143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:57:03.099640 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:03.100468 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:03.101895 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:03.102590 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:03.102698 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:03.102862 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0929 17:57:03.103030 (+ 168us) server_negotiation.cc:299] Negotiation successful
0929 17:57:03.103132 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":260,"thread_start_us":170,"threads_started":1}
I20260929 17:57:03.103281 20071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:03.097758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37641 (local address 127.0.0.1:47560)
0929 17:57:03.098060 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0929 17:57:03.098121 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0929 17:57:03.098256 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:57:03.099445 (+ 1189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:57:03.099448 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:57:03.099464 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:57:03.099555 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:57:03.099561 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:57:03.100583 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:57:03.100586 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:57:03.101783 (+ 1197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:57:03.101792 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:57:03.102717 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:57:03.102721 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:57:03.102827 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:03.102870 (+ 43us) client_negotiation.cc:769] Sending connection context
0929 17:57:03.102987 (+ 117us) client_negotiation.cc:240] Negotiation successful
0929 17:57:03.103117 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186,"spinlock_wait_cycles":7680,"thread_start_us":100,"threads_started":1}
I20260929 17:57:03.103731 20072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:03.098240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38211 (local address 127.0.0.1:51556)
0929 17:57:03.099208 (+ 968us) negotiation.cc:107] Waiting for socket to connect
0929 17:57:03.099222 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 17:57:03.099304 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:57:03.099733 (+ 429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:57:03.099735 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:57:03.099752 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:57:03.099838 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:57:03.099844 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:57:03.101135 (+ 1291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:57:03.101139 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:57:03.102265 (+ 1126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:57:03.102272 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:57:03.103356 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:57:03.103359 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:57:03.103456 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:03.103479 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 17:57:03.103542 (+ 63us) client_negotiation.cc:240] Negotiation successful
0929 17:57:03.103623 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":895,"mutex_wait_us":2,"thread_start_us":87,"threads_started":1}
I20260929 17:57:03.104147 20074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:03.098820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51556 (local address 127.0.98.2:38211)
0929 17:57:03.099311 (+ 491us) server_negotiation.cc:206] Beginning negotiation
0929 17:57:03.099323 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 17:57:03.099347 (+ 24us) server_negotiation.cc:407] Connection header received
0929 17:57:03.099418 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:57:03.099423 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:57:03.099609 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:57:03.099764 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:57:03.099914 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:03.100934 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:03.102378 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:03.103060 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:03.103149 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:03.103295 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0929 17:57:03.103895 (+ 600us) server_negotiation.cc:299] Negotiation successful
0929 17:57:03.103999 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":399,"thread_start_us":111,"threads_started":1}
I20260929 17:57:03.104219 20069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:03.097436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35791 (local address 127.0.0.1:42498)
0929 17:57:03.098060 (+ 624us) negotiation.cc:107] Waiting for socket to connect
0929 17:57:03.098121 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0929 17:57:03.098271 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:57:03.099098 (+ 827us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:57:03.099104 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:57:03.099130 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:57:03.099281 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:57:03.099287 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:57:03.100789 (+ 1502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:57:03.100792 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:57:03.102067 (+ 1275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:57:03.102077 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:57:03.103289 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:57:03.103293 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:57:03.103944 (+ 651us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:03.103968 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 17:57:03.104032 (+ 64us) client_negotiation.cc:240] Negotiation successful
0929 17:57:03.104102 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":209,"mutex_wait_us":32,"spinlock_wait_cycles":2560,"thread_start_us":125,"threads_started":1}
I20260929 17:57:03.104938 20070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:03.097540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42498 (local address 127.0.98.1:35791)
0929 17:57:03.098458 (+ 918us) server_negotiation.cc:206] Beginning negotiation
0929 17:57:03.098476 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 17:57:03.098502 (+ 26us) server_negotiation.cc:407] Connection header received
0929 17:57:03.098581 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:57:03.098587 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:57:03.098839 (+ 252us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:57:03.099003 (+ 164us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:57:03.099424 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:03.100643 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:03.102199 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:03.102982 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:03.103082 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:03.103215 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0929 17:57:03.104084 (+ 869us) server_negotiation.cc:299] Negotiation successful
0929 17:57:03.104792 (+ 708us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":803,"thread_start_us":77,"threads_started":1}
I20260929 17:57:03.106590 19992 tablet_service.cc:1467] Processing CreateTablet for tablet 5e09998096974e718e7b70ff00d3a63a (DEFAULT_TABLE table=test-table [id=90cd94bedd1145c6a9e8a85c774d6666]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 17:57:03.107095 19858 tablet_service.cc:1467] Processing CreateTablet for tablet 5e09998096974e718e7b70ff00d3a63a (DEFAULT_TABLE table=test-table [id=90cd94bedd1145c6a9e8a85c774d6666]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 17:57:03.107563 19992 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e09998096974e718e7b70ff00d3a63a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:57:03.107575 19710 tablet_service.cc:1467] Processing CreateTablet for tablet 5e09998096974e718e7b70ff00d3a63a (DEFAULT_TABLE table=test-table [id=90cd94bedd1145c6a9e8a85c774d6666]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 17:57:03.108186 19858 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e09998096974e718e7b70ff00d3a63a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:57:03.108552 19710 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e09998096974e718e7b70ff00d3a63a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:57:03.114374 20075 tablet_bootstrap.cc:492] T 5e09998096974e718e7b70ff00d3a63a P 0af7d6cca2d7458cbfeca8952d007dab: Bootstrap starting.
I20260929 17:57:03.115280 20076 tablet_bootstrap.cc:492] T 5e09998096974e718e7b70ff00d3a63a P 4e6e3f80fc904b92854863c35b3fe1aa: Bootstrap starting.
I20260929 17:57:03.116225 20077 tablet_bootstrap.cc:492] T 5e09998096974e718e7b70ff00d3a63a P 85d41a99a01847c092aef2c87ee7affc: Bootstrap starting.
I20260929 17:57:03.116771 20075 tablet_bootstrap.cc:654] T 5e09998096974e718e7b70ff00d3a63a P 0af7d6cca2d7458cbfeca8952d007dab: Neither blocks nor log segments found. Creating new log.
I20260929 17:57:03.117457 20076 tablet_bootstrap.cc:654] T 5e09998096974e718e7b70ff00d3a63a P 4e6e3f80fc904b92854863c35b3fe1aa: Neither blocks nor log segments found. Creating new log.
I20260929 17:57:03.117947 20077 tablet_bootstrap.cc:654] T 5e09998096974e718e7b70ff00d3a63a P 85d41a99a01847c092aef2c87ee7affc: Neither blocks nor log segments found. Creating new log.
I20260929 17:57:03.118106 20075 log.cc:824] T 5e09998096974e718e7b70ff00d3a63a P 0af7d6cca2d7458cbfeca8952d007dab: Log is configured to *not* fsync() on all Append() calls
I20260929 17:57:03.118296 20076 log.cc:824] T 5e09998096974e718e7b70ff00d3a63a P 4e6e3f80fc904b92854863c35b3fe1aa: Log is configured to *not* fsync() on all Append() calls
I20260929 17:57:03.119052 20077 log.cc:824] T 5e09998096974e718e7b70ff00d3a63a P 85d41a99a01847c092aef2c87ee7affc: Log is configured to *not* fsync() on all Append() calls
I20260929 17:57:03.120293 20075 tablet_bootstrap.cc:492] T 5e09998096974e718e7b70ff00d3a63a P 0af7d6cca2d7458cbfeca8952d007dab: No bootstrap required, opened a new log
I20260929 17:57:03.120464 20076 tablet_bootstrap.cc:492] T 5e09998096974e718e7b70ff00d3a63a P 4e6e3f80fc904b92854863c35b3fe1aa: No bootstrap required, opened a new log
I20260929 17:57:03.120499 20075 ts_tablet_manager.cc:1397] T 5e09998096974e718e7b70ff00d3a63a P 0af7d6cca2d7458cbfeca8952d007dab: Time spent bootstrapping tablet: real 0.006s user 0.006s sys 0.000s
I20260929 17:57:03.120604 20076 ts_tablet_manager.cc:1397] T 5e09998096974e718e7b70ff00d3a63a P 4e6e3f80fc904b92854863c35b3fe1aa: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 17:57:03.121307 20077 tablet_bootstrap.cc:492] T 5e09998096974e718e7b70ff00d3a63a P 85d41a99a01847c092aef2c87ee7affc: No bootstrap required, opened a new log
I20260929 17:57:03.121493 20077 ts_tablet_manager.cc:1397] T 5e09998096974e718e7b70ff00d3a63a P 85d41a99a01847c092aef2c87ee7affc: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260929 17:57:03.124063 20076 raft_consensus.cc:348] T 5e09998096974e718e7b70ff00d3a63a P 4e6e3f80fc904b92854863c35b3fe1aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e6e3f80fc904b92854863c35b3fe1aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35791 } } peers { permanent_uuid: "85d41a99a01847c092aef2c87ee7affc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38211 } } peers { permanent_uuid: "0af7d6cca2d7458cbfeca8952d007dab" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37641 } }
I20260929 17:57:03.124313 20076 raft_consensus.cc:374] T 5e09998096974e718e7b70ff00d3a63a P 4e6e3f80fc904b92854863c35b3fe1aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:57:03.124385 20076 raft_consensus.cc:729] T 5e09998096974e718e7b70ff00d3a63a P 4e6e3f80fc904b92854863c35b3fe1aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e6e3f80fc904b92854863c35b3fe1aa, State: Initialized, Role: FOLLOWER
I20260929 17:57:03.124856 20076 consensus_queue.cc:260] T 5e09998096974e718e7b70ff00d3a63a P 4e6e3f80fc904b92854863c35b3fe1aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e6e3f80fc904b92854863c35b3fe1aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35791 } } peers { permanent_uuid: "85d41a99a01847c092aef2c87ee7affc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38211 } } peers { permanent_uuid: "0af7d6cca2d7458cbfeca8952d007dab" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37641 } }
I20260929 17:57:03.124739 20075 raft_consensus.cc:348] T 5e09998096974e718e7b70ff00d3a63a P 0af7d6cca2d7458cbfeca8952d007dab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e6e3f80fc904b92854863c35b3fe1aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35791 } } peers { permanent_uuid: "85d41a99a01847c092aef2c87ee7affc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38211 } } peers { permanent_uuid: "0af7d6cca2d7458cbfeca8952d007dab" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37641 } }
I20260929 17:57:03.125034 20075 raft_consensus.cc:374] T 5e09998096974e718e7b70ff00d3a63a P 0af7d6cca2d7458cbfeca8952d007dab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:57:03.125118 20075 raft_consensus.cc:729] T 5e09998096974e718e7b70ff00d3a63a P 0af7d6cca2d7458cbfeca8952d007dab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0af7d6cca2d7458cbfeca8952d007dab, State: Initialized, Role: FOLLOWER
I20260929 17:57:03.125247 20077 raft_consensus.cc:348] T 5e09998096974e718e7b70ff00d3a63a P 85d41a99a01847c092aef2c87ee7affc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e6e3f80fc904b92854863c35b3fe1aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35791 } } peers { permanent_uuid: "85d41a99a01847c092aef2c87ee7affc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38211 } } peers { permanent_uuid: "0af7d6cca2d7458cbfeca8952d007dab" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37641 } }
I20260929 17:57:03.125519 20077 raft_consensus.cc:374] T 5e09998096974e718e7b70ff00d3a63a P 85d41a99a01847c092aef2c87ee7affc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:57:03.125576 19790 heartbeater.cc:503] Master 127.0.98.62:40557 was elected leader, sending a full tablet report...
I20260929 17:57:03.125586 20077 raft_consensus.cc:729] T 5e09998096974e718e7b70ff00d3a63a P 85d41a99a01847c092aef2c87ee7affc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85d41a99a01847c092aef2c87ee7affc, State: Initialized, Role: FOLLOWER
I20260929 17:57:03.125567 20075 consensus_queue.cc:260] T 5e09998096974e718e7b70ff00d3a63a P 0af7d6cca2d7458cbfeca8952d007dab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e6e3f80fc904b92854863c35b3fe1aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35791 } } peers { permanent_uuid: "85d41a99a01847c092aef2c87ee7affc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38211 } } peers { permanent_uuid: "0af7d6cca2d7458cbfeca8952d007dab" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37641 } }
I20260929 17:57:03.126042 20077 consensus_queue.cc:260] T 5e09998096974e718e7b70ff00d3a63a P 85d41a99a01847c092aef2c87ee7affc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e6e3f80fc904b92854863c35b3fe1aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35791 } } peers { permanent_uuid: "85d41a99a01847c092aef2c87ee7affc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38211 } } peers { permanent_uuid: "0af7d6cca2d7458cbfeca8952d007dab" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37641 } }
I20260929 17:57:03.126611 20075 ts_tablet_manager.cc:1428] T 5e09998096974e718e7b70ff00d3a63a P 0af7d6cca2d7458cbfeca8952d007dab: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260929 17:57:03.126955 20077 ts_tablet_manager.cc:1428] T 5e09998096974e718e7b70ff00d3a63a P 85d41a99a01847c092aef2c87ee7affc: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260929 17:57:03.126959 20058 heartbeater.cc:503] Master 127.0.98.62:40557 was elected leader, sending a full tablet report...
I20260929 17:57:03.127102 20076 ts_tablet_manager.cc:1428] T 5e09998096974e718e7b70ff00d3a63a P 4e6e3f80fc904b92854863c35b3fe1aa: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260929 17:57:03.127292 19924 heartbeater.cc:503] Master 127.0.98.62:40557 was elected leader, sending a full tablet report...
I20260929 17:57:03.175345 20083 raft_consensus.cc:482] T 5e09998096974e718e7b70ff00d3a63a P 85d41a99a01847c092aef2c87ee7affc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 17:57:03.175536 20083 raft_consensus.cc:504] T 5e09998096974e718e7b70ff00d3a63a P 85d41a99a01847c092aef2c87ee7affc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e6e3f80fc904b92854863c35b3fe1aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35791 } } peers { permanent_uuid: "85d41a99a01847c092aef2c87ee7affc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38211 } } peers { permanent_uuid: "0af7d6cca2d7458cbfeca8952d007dab" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37641 } }
I20260929 17:57:03.176661 20083 leader_election.cc:288] T 5e09998096974e718e7b70ff00d3a63a P 85d41a99a01847c092aef2c87ee7affc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4e6e3f80fc904b92854863c35b3fe1aa (127.0.98.1:35791), 0af7d6cca2d7458cbfeca8952d007dab (127.0.98.3:37641)
I20260929 17:57:03.181702 20070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:03.177545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38259 (local address 127.0.98.1:35791)
0929 17:57:03.177677 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0929 17:57:03.177682 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:57:03.177695 (+ 13us) server_negotiation.cc:407] Connection header received
0929 17:57:03.177760 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:57:03.177763 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:57:03.177805 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:57:03.177867 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:57:03.178114 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:03.178843 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:03.180398 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:03.181089 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:03.181180 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:03.181246 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0929 17:57:03.181498 (+ 252us) server_negotiation.cc:299] Negotiation successful
0929 17:57:03.181568 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 17:57:03.181701 20073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:03.177068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59039 (local address 127.0.98.3:37641)
0929 17:57:03.177223 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0929 17:57:03.177227 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:57:03.177368 (+ 141us) server_negotiation.cc:407] Connection header received
0929 17:57:03.177506 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:57:03.177510 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:57:03.177565 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:57:03.177632 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:57:03.177913 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:03.178702 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:03.180397 (+ 1695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:03.181072 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:03.181180 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:03.181257 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0929 17:57:03.181498 (+ 241us) server_negotiation.cc:299] Negotiation successful
0929 17:57:03.181568 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 17:57:03.181741 19926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:03.176961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37641 (local address 127.0.98.2:59039)
0929 17:57:03.177300 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0929 17:57:03.177312 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 17:57:03.177399 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:57:03.177671 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:57:03.177683 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:57:03.177698 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:57:03.177805 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:57:03.177810 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:57:03.178839 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:57:03.178842 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:57:03.180255 (+ 1413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:57:03.180267 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:57:03.181220 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:57:03.181224 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:57:03.181345 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:03.181401 (+ 56us) client_negotiation.cc:769] Sending connection context
0929 17:57:03.181482 (+ 81us) client_negotiation.cc:240] Negotiation successful
0929 17:57:03.181572 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"mutex_wait_us":68}
I20260929 17:57:03.181749 20084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:03.177026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35791 (local address 127.0.98.2:38259)
0929 17:57:03.177351 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0929 17:57:03.177365 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 17:57:03.177463 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:57:03.177900 (+ 437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:57:03.177902 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:57:03.177918 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:57:03.178008 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:57:03.178012 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:57:03.178951 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:57:03.178953 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:57:03.180259 (+ 1306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:57:03.180267 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:57:03.181217 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:57:03.181221 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:57:03.181342 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:03.181400 (+ 58us) client_negotiation.cc:769] Sending connection context
0929 17:57:03.181482 (+ 82us) client_negotiation.cc:240] Negotiation successful
0929 17:57:03.181572 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"mutex_wait_us":73,"thread_start_us":155,"threads_started":1}
I20260929 17:57:03.182377 19744 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e09998096974e718e7b70ff00d3a63a" candidate_uuid: "85d41a99a01847c092aef2c87ee7affc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e6e3f80fc904b92854863c35b3fe1aa" is_pre_election: true
I20260929 17:57:03.182399 20012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e09998096974e718e7b70ff00d3a63a" candidate_uuid: "85d41a99a01847c092aef2c87ee7affc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0af7d6cca2d7458cbfeca8952d007dab" is_pre_election: true
I20260929 17:57:03.182660 20012 raft_consensus.cc:2454] T 5e09998096974e718e7b70ff00d3a63a P 0af7d6cca2d7458cbfeca8952d007dab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85d41a99a01847c092aef2c87ee7affc in term 0.
I20260929 17:57:03.182722 19744 raft_consensus.cc:2454] T 5e09998096974e718e7b70ff00d3a63a P 4e6e3f80fc904b92854863c35b3fe1aa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85d41a99a01847c092aef2c87ee7affc in term 0.
I20260929 17:57:03.183176 19814 leader_election.cc:302] T 5e09998096974e718e7b70ff00d3a63a P 85d41a99a01847c092aef2c87ee7affc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0af7d6cca2d7458cbfeca8952d007dab, 85d41a99a01847c092aef2c87ee7affc; no voters:
I20260929 17:57:03.183444 20083 raft_consensus.cc:2781] T 5e09998096974e718e7b70ff00d3a63a P 85d41a99a01847c092aef2c87ee7affc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 17:57:03.183545 20083 raft_consensus.cc:482] T 5e09998096974e718e7b70ff00d3a63a P 85d41a99a01847c092aef2c87ee7affc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 17:57:03.183597 20083 raft_consensus.cc:3037] T 5e09998096974e718e7b70ff00d3a63a P 85d41a99a01847c092aef2c87ee7affc [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:57:03.184680 20083 raft_consensus.cc:504] T 5e09998096974e718e7b70ff00d3a63a P 85d41a99a01847c092aef2c87ee7affc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e6e3f80fc904b92854863c35b3fe1aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35791 } } peers { permanent_uuid: "85d41a99a01847c092aef2c87ee7affc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38211 } } peers { permanent_uuid: "0af7d6cca2d7458cbfeca8952d007dab" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37641 } }
I20260929 17:57:03.185047 20083 leader_election.cc:288] T 5e09998096974e718e7b70ff00d3a63a P 85d41a99a01847c092aef2c87ee7affc [CANDIDATE]: Term 1 election: Requested vote from peers 4e6e3f80fc904b92854863c35b3fe1aa (127.0.98.1:35791), 0af7d6cca2d7458cbfeca8952d007dab (127.0.98.3:37641)
I20260929 17:57:03.185422 20012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e09998096974e718e7b70ff00d3a63a" candidate_uuid: "85d41a99a01847c092aef2c87ee7affc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0af7d6cca2d7458cbfeca8952d007dab"
I20260929 17:57:03.185473 19744 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e09998096974e718e7b70ff00d3a63a" candidate_uuid: "85d41a99a01847c092aef2c87ee7affc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e6e3f80fc904b92854863c35b3fe1aa"
I20260929 17:57:03.185561 20012 raft_consensus.cc:3037] T 5e09998096974e718e7b70ff00d3a63a P 0af7d6cca2d7458cbfeca8952d007dab [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:57:03.185578 19744 raft_consensus.cc:3037] T 5e09998096974e718e7b70ff00d3a63a P 4e6e3f80fc904b92854863c35b3fe1aa [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:57:03.186905 19744 raft_consensus.cc:2454] T 5e09998096974e718e7b70ff00d3a63a P 4e6e3f80fc904b92854863c35b3fe1aa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85d41a99a01847c092aef2c87ee7affc in term 1.
I20260929 17:57:03.186905 20012 raft_consensus.cc:2454] T 5e09998096974e718e7b70ff00d3a63a P 0af7d6cca2d7458cbfeca8952d007dab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85d41a99a01847c092aef2c87ee7affc in term 1.
I20260929 17:57:03.187213 19811 leader_election.cc:302] T 5e09998096974e718e7b70ff00d3a63a P 85d41a99a01847c092aef2c87ee7affc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e6e3f80fc904b92854863c35b3fe1aa, 85d41a99a01847c092aef2c87ee7affc; no voters:
I20260929 17:57:03.187439 20083 raft_consensus.cc:2781] T 5e09998096974e718e7b70ff00d3a63a P 85d41a99a01847c092aef2c87ee7affc [term 1 FOLLOWER]: Leader election won for term 1
I20260929 17:57:03.187664 20083 raft_consensus.cc:686] T 5e09998096974e718e7b70ff00d3a63a P 85d41a99a01847c092aef2c87ee7affc [term 1 LEADER]: Becoming Leader. State: Replica: 85d41a99a01847c092aef2c87ee7affc, State: Running, Role: LEADER
I20260929 17:57:03.187969 20083 consensus_queue.cc:237] T 5e09998096974e718e7b70ff00d3a63a P 85d41a99a01847c092aef2c87ee7affc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e6e3f80fc904b92854863c35b3fe1aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35791 } } peers { permanent_uuid: "85d41a99a01847c092aef2c87ee7affc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38211 } } peers { permanent_uuid: "0af7d6cca2d7458cbfeca8952d007dab" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37641 } }
I20260929 17:57:03.191250 19599 catalog_manager.cc:5495] T 5e09998096974e718e7b70ff00d3a63a P 85d41a99a01847c092aef2c87ee7affc reported cstate change: term changed from 0 to 1, leader changed from <none> to 85d41a99a01847c092aef2c87ee7affc (127.0.98.2). New cstate: current_term: 1 leader_uuid: "85d41a99a01847c092aef2c87ee7affc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e6e3f80fc904b92854863c35b3fe1aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35791 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "85d41a99a01847c092aef2c87ee7affc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38211 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0af7d6cca2d7458cbfeca8952d007dab" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37641 } health_report { overall_health: UNKNOWN } } }
I20260929 17:57:03.202395 20074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:03.198937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51564 (local address 127.0.98.2:38211)
0929 17:57:03.199056 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0929 17:57:03.199060 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:57:03.199073 (+ 13us) server_negotiation.cc:407] Connection header received
0929 17:57:03.199229 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:57:03.199232 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:57:03.199284 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:57:03.199374 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 17:57:03.199592 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:03.200102 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:03.200954 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:03.201216 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:03.201303 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:03.201581 (+ 278us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 17:57:03.202015 (+ 434us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 17:57:03.202118 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0929 17:57:03.202183 (+ 65us) server_negotiation.cc:299] Negotiation successful
0929 17:57:03.202240 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260929 17:57:03.208405 20012 raft_consensus.cc:1261] T 5e09998096974e718e7b70ff00d3a63a P 0af7d6cca2d7458cbfeca8952d007dab [term 1 FOLLOWER]: Refusing update from remote peer 85d41a99a01847c092aef2c87ee7affc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 17:57:03.208405 19744 raft_consensus.cc:1261] T 5e09998096974e718e7b70ff00d3a63a P 4e6e3f80fc904b92854863c35b3fe1aa [term 1 FOLLOWER]: Refusing update from remote peer 85d41a99a01847c092aef2c87ee7affc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 17:57:03.209203 20083 consensus_queue.cc:1035] T 5e09998096974e718e7b70ff00d3a63a P 85d41a99a01847c092aef2c87ee7affc [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e6e3f80fc904b92854863c35b3fe1aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 17:57:03.209429 20085 consensus_queue.cc:1035] T 5e09998096974e718e7b70ff00d3a63a P 85d41a99a01847c092aef2c87ee7affc [LEADER]: Connected to new peer: Peer: permanent_uuid: "0af7d6cca2d7458cbfeca8952d007dab" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37641 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 17:57:03.227397 19598 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56716:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 17:57:03.227596 19598 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56716:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 17:57:03.230304 19598 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9fbe87dbf95e44bea93bac28f7df4e06: Sending DeleteTablet for 3 replicas of tablet 5e09998096974e718e7b70ff00d3a63a
I20260929 17:57:03.231132 19710 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e09998096974e718e7b70ff00d3a63a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 17:57:03 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42498
I20260929 17:57:03.231223 19992 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e09998096974e718e7b70ff00d3a63a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 17:57:03 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47560
I20260929 17:57:03.231290 19858 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e09998096974e718e7b70ff00d3a63a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 17:57:03 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51556
I20260929 17:57:03.231722 20099 tablet_replica.cc:330] T 5e09998096974e718e7b70ff00d3a63a P 85d41a99a01847c092aef2c87ee7affc: stopping tablet replica
I20260929 17:57:03.231725 20100 tablet_replica.cc:330] T 5e09998096974e718e7b70ff00d3a63a P 0af7d6cca2d7458cbfeca8952d007dab: stopping tablet replica
I20260929 17:57:03.231982 20098 tablet_replica.cc:330] T 5e09998096974e718e7b70ff00d3a63a P 4e6e3f80fc904b92854863c35b3fe1aa: stopping tablet replica
I20260929 17:57:03.232084 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskup3Soe/build/debug/bin/kudu with pid 19661
I20260929 17:57:03.232149 20099 raft_consensus.cc:2229] T 5e09998096974e718e7b70ff00d3a63a P 85d41a99a01847c092aef2c87ee7affc [term 1 LEADER]: Raft consensus shutting down.
I20260929 17:57:03.233325 20099 raft_consensus.cc:2258] T 5e09998096974e718e7b70ff00d3a63a P 85d41a99a01847c092aef2c87ee7affc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 17:57:03.234602 20100 raft_consensus.cc:2229] T 5e09998096974e718e7b70ff00d3a63a P 0af7d6cca2d7458cbfeca8952d007dab [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 17:57:03.234946 20099 ts_tablet_manager.cc:1905] T 5e09998096974e718e7b70ff00d3a63a P 85d41a99a01847c092aef2c87ee7affc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 17:57:03.234992 20100 raft_consensus.cc:2258] T 5e09998096974e718e7b70ff00d3a63a P 0af7d6cca2d7458cbfeca8952d007dab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 17:57:03.238168 20099 ts_tablet_manager.cc:1918] T 5e09998096974e718e7b70ff00d3a63a P 85d41a99a01847c092aef2c87ee7affc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 17:57:03.238350 20099 log.cc:1196] T 5e09998096974e718e7b70ff00d3a63a P 85d41a99a01847c092aef2c87ee7affc: Deleting WAL directory at /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704375975685-392-0/minicluster-data/ts-1/wal/wals/5e09998096974e718e7b70ff00d3a63a
I20260929 17:57:03.238559 20100 ts_tablet_manager.cc:1905] T 5e09998096974e718e7b70ff00d3a63a P 0af7d6cca2d7458cbfeca8952d007dab: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260929 17:57:03.238782 19583 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:35791 (error 108)
I20260929 17:57:03.239660 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskup3Soe/build/debug/bin/kudu with pid 19795
I20260929 17:57:03.241199 20100 ts_tablet_manager.cc:1918] T 5e09998096974e718e7b70ff00d3a63a P 0af7d6cca2d7458cbfeca8952d007dab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 17:57:03.241297 20100 log.cc:1196] T 5e09998096974e718e7b70ff00d3a63a P 0af7d6cca2d7458cbfeca8952d007dab: Deleting WAL directory at /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790704375975685-392-0/minicluster-data/ts-2/wal/wals/5e09998096974e718e7b70ff00d3a63a
I20260929 17:57:03.241605 20100 ts_tablet_manager.cc:1939] T 5e09998096974e718e7b70ff00d3a63a P 0af7d6cca2d7458cbfeca8952d007dab: Deleting consensus metadata
I20260929 17:57:03.242391 20069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:03.242028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35791 (local address 127.0.0.1:42500)
0929 17:57:03.242246 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0929 17:57:03.242312 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121}
W20260929 17:57:03.242578 19583 catalog_manager.cc:4584] TS 4e6e3f80fc904b92854863c35b3fe1aa (127.0.98.1:35791): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5e09998096974e718e7b70ff00d3a63a: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35791: connect: Connection refused (error 111)
I20260929 17:57:03.243476 19586 catalog_manager.cc:4841] TS 0af7d6cca2d7458cbfeca8952d007dab (127.0.98.3:37641): tablet 5e09998096974e718e7b70ff00d3a63a (table test-table [id=90cd94bedd1145c6a9e8a85c774d6666]) successfully deleted
I20260929 17:57:03.245929 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskup3Soe/build/debug/bin/kudu with pid 19929
I20260929 17:57:03.246732 20069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:03.246096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38211 (local address 127.0.0.1:51576)
0929 17:57:03.246585 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0929 17:57:03.246641 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:38211: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":407}
I20260929 17:57:03.251293 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskup3Soe/build/debug/bin/kudu with pid 19567
2026-09-29T17:57:03Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2198 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-29T17:57:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T17:57:03Z Disabled control of system clock
I20260929 17:57:03.280179 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskup3Soe/build/debug/bin/kudu
/tmp/dist-test-taskup3Soe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790704375975685-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790704375975685-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790704375975685-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790704375975685-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36629
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:39903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36629
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 17:57:03.393172 20105 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:57:03.393452 20105 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:57:03.393489 20105 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:57:03.397564 20105 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 17:57:03.397629 20105 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:57:03.397642 20105 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:57:03.397658 20105 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 17:57:03.397672 20105 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 17:57:03.402455 20105 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790704375975685-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790704375975685-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36629
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36629
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790704375975685-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20105
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790704375975685-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:57:03.403513 20105 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:57:03.404844 20105 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:57:03.413859 20113 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:57:03.413849 20110 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:57:03.413846 20111 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:57:03.414283 20105 server_base.cc:1034] running on GCE node
I20260929 17:57:03.414951 20105 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:57:03.416167 20105 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:57:03.417338 20105 hybrid_clock.cc:648] HybridClock initialized: now 1790704623417315 us; error 42 us; skew 500 ppm
I20260929 17:57:03.419366 20105 webserver.cc:460] Webserver started at http://127.0.98.62:36595/ using document root <none> and password file <none>
I20260929 17:57:03.419953 20105 fs_manager.cc:359] Metadata directory not provided
I20260929 17:57:03.420025 20105 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:57:03.420244 20105 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:57:03.422487 20105 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790704375975685-392-0/minicluster-data/master-0/data/instance:
uuid: "02683cfb36704a498d3a87fa622236e4"
format_stamp: "Formatted at 2026-09-29 17:57:03 on dist-test-slave-53nh"
server_key: "0349d6e6f7a3d1685431fb8559e89d0c"
server_key_iv: "4abb84623a8d46d92878f7f6f8f942fd"
server_key_version: "encryptionkey@0"
I20260929 17:57:03.423065 20105 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790704375975685-392-0/minicluster-data/master-0/wal/instance:
uuid: "02683cfb36704a498d3a87fa622236e4"
format_stamp: "Formatted at 2026-09-29 17:57:03 on dist-test-slave-53nh"
server_key: "0349d6e6f7a3d1685431fb8559e89d0c"
server_key_iv: "4abb84623a8d46d92878f7f6f8f942fd"
server_key_version: "encryptionkey@0"
I20260929 17:57:03.427012 20105 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 17:57:03.429643 20119 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:57:03.430931 20105 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 17:57:03.431057 20105 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790704375975685-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790704375975685-392-0/minicluster-data/master-0/wal
uuid: "02683cfb36704a498d3a87fa622236e4"
format_stamp: "Formatted at 2026-09-29 17:57:03 on dist-test-slave-53nh"
server_key: "0349d6e6f7a3d1685431fb8559e89d0c"
server_key_iv: "4abb84623a8d46d92878f7f6f8f942fd"
server_key_version: "encryptionkey@0"
I20260929 17:57:03.431172 20105 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790704375975685-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790704375975685-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790704375975685-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:57:03.470188 20105 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:57:03.470944 20105 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:57:03.471128 20105 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:57:03.477890 20105 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36629
I20260929 17:57:03.477952 20171 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36629 every 8 connection(s)
I20260929 17:57:03.478854 20105 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790704375975685-392-0/minicluster-data/master-0/data/info.pb
I20260929 17:57:03.482014 20172 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:57:03.486424 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskup3Soe/build/debug/bin/kudu as pid 20105
I20260929 17:57:03.486531 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790704375975685-392-0/minicluster-data/master-0/wal/instance
I20260929 17:57:03.486822 392 external_mini_cluster.cc:1428] Setting key 2963fcccdd89fb427e1bd1af73c2b726
I20260929 17:57:03.487901 20172 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 02683cfb36704a498d3a87fa622236e4: Bootstrap starting.
I20260929 17:57:03.490470 20172 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 02683cfb36704a498d3a87fa622236e4: Neither blocks nor log segments found. Creating new log.
I20260929 17:57:03.491246 20172 log.cc:824] T 00000000000000000000000000000000 P 02683cfb36704a498d3a87fa622236e4: Log is configured to *not* fsync() on all Append() calls
I20260929 17:57:03.493278 20172 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 02683cfb36704a498d3a87fa622236e4: No bootstrap required, opened a new log
I20260929 17:57:03.496486 20175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:03.488299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60144 (local address 127.0.98.62:36629)
0929 17:57:03.489028 (+ 729us) server_negotiation.cc:206] Beginning negotiation
0929 17:57:03.489050 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0929 17:57:03.489091 (+ 41us) server_negotiation.cc:407] Connection header received
0929 17:57:03.489800 (+ 709us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:57:03.489820 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:57:03.490275 (+ 455us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:57:03.490608 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:57:03.491070 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:03.492074 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:03.492973 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:03.493536 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:03.493686 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:03.494616 (+ 930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:57:03.494621 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:57:03.494636 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 17:57:03.495068 (+ 432us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:57:03.495130 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 17:57:03.495151 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:57:03.495266 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0929 17:57:03.495411 (+ 145us) server_negotiation.cc:299] Negotiation successful
0929 17:57:03.495677 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298,"thread_start_us":138,"threads_started":1}
I20260929 17:57:03.496943 20172 raft_consensus.cc:348] T 00000000000000000000000000000000 P 02683cfb36704a498d3a87fa622236e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02683cfb36704a498d3a87fa622236e4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36629 } }
I20260929 17:57:03.497232 20172 raft_consensus.cc:374] T 00000000000000000000000000000000 P 02683cfb36704a498d3a87fa622236e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:57:03.497299 20172 raft_consensus.cc:729] T 00000000000000000000000000000000 P 02683cfb36704a498d3a87fa622236e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02683cfb36704a498d3a87fa622236e4, State: Initialized, Role: FOLLOWER
I20260929 17:57:03.497748 20172 consensus_queue.cc:260] T 00000000000000000000000000000000 P 02683cfb36704a498d3a87fa622236e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02683cfb36704a498d3a87fa622236e4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36629 } }
I20260929 17:57:03.497908 20172 raft_consensus.cc:388] T 00000000000000000000000000000000 P 02683cfb36704a498d3a87fa622236e4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 17:57:03.497970 20172 raft_consensus.cc:482] T 00000000000000000000000000000000 P 02683cfb36704a498d3a87fa622236e4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 17:57:03.498061 20172 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 02683cfb36704a498d3a87fa622236e4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:57:03.499224 20172 raft_consensus.cc:504] T 00000000000000000000000000000000 P 02683cfb36704a498d3a87fa622236e4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02683cfb36704a498d3a87fa622236e4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36629 } }
I20260929 17:57:03.499529 20172 leader_election.cc:302] T 00000000000000000000000000000000 P 02683cfb36704a498d3a87fa622236e4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 02683cfb36704a498d3a87fa622236e4; no voters:
I20260929 17:57:03.499801 20172 leader_election.cc:288] T 00000000000000000000000000000000 P 02683cfb36704a498d3a87fa622236e4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 17:57:03.499909 20177 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 02683cfb36704a498d3a87fa622236e4 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 17:57:03.500092 20177 raft_consensus.cc:686] T 00000000000000000000000000000000 P 02683cfb36704a498d3a87fa622236e4 [term 1 LEADER]: Becoming Leader. State: Replica: 02683cfb36704a498d3a87fa622236e4, State: Running, Role: LEADER
I20260929 17:57:03.500391 20177 consensus_queue.cc:237] T 00000000000000000000000000000000 P 02683cfb36704a498d3a87fa622236e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02683cfb36704a498d3a87fa622236e4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36629 } }
I20260929 17:57:03.500927 20172 sys_catalog.cc:564] T 00000000000000000000000000000000 P 02683cfb36704a498d3a87fa622236e4 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 17:57:03.502018 20179 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02683cfb36704a498d3a87fa622236e4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 02683cfb36704a498d3a87fa622236e4. Latest consensus state: current_term: 1 leader_uuid: "02683cfb36704a498d3a87fa622236e4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02683cfb36704a498d3a87fa622236e4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36629 } } }
I20260929 17:57:03.502148 20179 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02683cfb36704a498d3a87fa622236e4 [sys.catalog]: This master's current role is: LEADER
I20260929 17:57:03.502632 20178 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02683cfb36704a498d3a87fa622236e4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "02683cfb36704a498d3a87fa622236e4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02683cfb36704a498d3a87fa622236e4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36629 } } }
I20260929 17:57:03.502997 20186 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 17:57:03.503273 20178 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02683cfb36704a498d3a87fa622236e4 [sys.catalog]: This master's current role is: LEADER
I20260929 17:57:03.505970 20186 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 17:57:03.511662 20186 catalog_manager.cc:1347] Generated new cluster ID: ac8dc930124d41f784b10affec80a71f
I20260929 17:57:03.511751 20186 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 17:57:03.518766 20186 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 17:57:03.519697 20186 catalog_manager.cc:1504] Loading token signing keys...
I20260929 17:57:03.530177 20186 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 02683cfb36704a498d3a87fa622236e4: Generated new TSK 0
I20260929 17:57:03.530949 20186 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 17:57:03.541301 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskup3Soe/build/debug/bin/kudu
/tmp/dist-test-taskup3Soe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790704375975685-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790704375975685-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790704375975685-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790704375975685-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36629
--builtin_ntp_servers=127.0.98.20:39903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 17:57:03.654132 20196 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:57:03.654443 20196 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:57:03.654484 20196 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:57:03.658671 20196 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:57:03.658747 20196 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:57:03.658811 20196 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 17:57:03.663830 20196 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790704375975685-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790704375975685-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790704375975685-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36629
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20196
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790704375975685-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:57:03.665024 20196 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:57:03.666499 20196 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 17:57:03.676081 20196 server_base.cc:1034] running on GCE node
W20260929 17:57:03.676088 20201 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:57:03.676076 20204 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:57:03.676313 20202 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:57:03.676710 20196 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:57:03.677397 20196 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:57:03.678561 20196 hybrid_clock.cc:648] HybridClock initialized: now 1790704623678540 us; error 35 us; skew 500 ppm
I20260929 17:57:03.680465 20196 webserver.cc:460] Webserver started at http://127.0.98.1:42903/ using document root <none> and password file <none>
I20260929 17:57:03.681063 20196 fs_manager.cc:359] Metadata directory not provided
I20260929 17:57:03.681134 20196 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:57:03.681332 20196 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:57:03.683344 20196 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790704375975685-392-0/minicluster-data/ts-0/data/instance:
uuid: "c5218c5d07194509bc7c6715cb028c11"
format_stamp: "Formatted at 2026-09-29 17:57:03 on dist-test-slave-53nh"
server_key: "7b0ed216d27f4e2345b230da7d5d1add"
server_key_iv: "f5efcbbadfeea643758ed85b635344af"
server_key_version: "encryptionkey@0"
I20260929 17:57:03.683882 20196 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790704375975685-392-0/minicluster-data/ts-0/wal/instance:
uuid: "c5218c5d07194509bc7c6715cb028c11"
format_stamp: "Formatted at 2026-09-29 17:57:03 on dist-test-slave-53nh"
server_key: "7b0ed216d27f4e2345b230da7d5d1add"
server_key_iv: "f5efcbbadfeea643758ed85b635344af"
server_key_version: "encryptionkey@0"
I20260929 17:57:03.687501 20196 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260929 17:57:03.690099 20210 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:57:03.691418 20196 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 17:57:03.691540 20196 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790704375975685-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790704375975685-392-0/minicluster-data/ts-0/wal
uuid: "c5218c5d07194509bc7c6715cb028c11"
format_stamp: "Formatted at 2026-09-29 17:57:03 on dist-test-slave-53nh"
server_key: "7b0ed216d27f4e2345b230da7d5d1add"
server_key_iv: "f5efcbbadfeea643758ed85b635344af"
server_key_version: "encryptionkey@0"
I20260929 17:57:03.691651 20196 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790704375975685-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790704375975685-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790704375975685-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:57:03.713552 20196 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:57:03.714303 20196 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:57:03.714483 20196 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:57:03.715102 20196 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 17:57:03.716077 20196 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:57:03.716126 20196 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:57:03.716179 20196 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:57:03.716205 20196 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 17:57:03.724920 20196 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42817
I20260929 17:57:03.724964 20323 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42817 every 8 connection(s)
I20260929 17:57:03.725898 20196 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790704375975685-392-0/minicluster-data/ts-0/data/info.pb
I20260929 17:57:03.727840 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskup3Soe/build/debug/bin/kudu as pid 20196
I20260929 17:57:03.727960 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790704375975685-392-0/minicluster-data/ts-0/wal/instance
I20260929 17:57:03.728225 392 external_mini_cluster.cc:1428] Setting key 5124f83cf85564096f981af0577730f7
I20260929 17:57:03.730214 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskup3Soe/build/debug/bin/kudu
/tmp/dist-test-taskup3Soe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790704375975685-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790704375975685-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790704375975685-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790704375975685-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36629
--builtin_ntp_servers=127.0.98.20:39903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 17:57:03.735852 20175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:03.727806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45503 (local address 127.0.98.62:36629)
0929 17:57:03.727971 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0929 17:57:03.727976 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:57:03.729222 (+ 1246us) server_negotiation.cc:407] Connection header received
0929 17:57:03.730753 (+ 1531us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:57:03.730756 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:57:03.730817 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:57:03.730892 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:57:03.731912 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:03.732462 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:03.733690 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:03.733931 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:03.734021 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:03.735290 (+ 1269us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:57:03.735293 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:57:03.735295 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 17:57:03.735316 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:57:03.735337 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 17:57:03.735346 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:57:03.735436 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0929 17:57:03.735614 (+ 178us) server_negotiation.cc:299] Negotiation successful
0929 17:57:03.735685 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 17:57:03.736825 20326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:03.728250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36629 (local address 127.0.98.1:45503)
0929 17:57:03.729059 (+ 809us) negotiation.cc:107] Waiting for socket to connect
0929 17:57:03.729119 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0929 17:57:03.730509 (+ 1390us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:57:03.731114 (+ 605us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:57:03.731122 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:57:03.731231 (+ 109us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:57:03.731764 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:57:03.731777 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:57:03.732590 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:57:03.732593 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:57:03.733565 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:57:03.733575 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:57:03.734081 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:57:03.734085 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:57:03.734258 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:03.734996 (+ 738us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 17:57:03.734998 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:57:03.735045 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 17:57:03.735119 (+ 74us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:57:03.735476 (+ 357us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:57:03.735484 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:57:03.735489 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 17:57:03.735601 (+ 112us) client_negotiation.cc:240] Negotiation successful
0929 17:57:03.735933 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":123,"threads_started":1}
I20260929 17:57:03.738220 20324 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36629
I20260929 17:57:03.738464 20324 heartbeater.cc:464] Registering TS with master...
I20260929 17:57:03.739032 20324 heartbeater.cc:511] Master 127.0.98.62:36629 requested a full tablet report, sending...
I20260929 17:57:03.740615 20136 ts_manager.cc:194] Registered new tserver with Master: c5218c5d07194509bc7c6715cb028c11 (127.0.98.1:42817)
I20260929 17:57:03.741849 20136 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:45503
W20260929 17:57:03.842784 20327 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:57:03.843055 20327 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:57:03.843092 20327 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:57:03.847259 20327 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:57:03.847334 20327 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:57:03.847397 20327 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 17:57:03.852464 20327 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790704375975685-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790704375975685-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790704375975685-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36629
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20327
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790704375975685-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:57:03.853632 20327 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:57:03.855141 20327 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 17:57:03.865084 20327 server_base.cc:1034] running on GCE node
W20260929 17:57:03.865043 20332 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:57:03.865043 20333 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:57:03.865043 20335 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:57:03.865738 20327 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:57:03.866430 20327 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:57:03.867604 20327 hybrid_clock.cc:648] HybridClock initialized: now 1790704623867580 us; error 37 us; skew 500 ppm
I20260929 17:57:03.869545 20327 webserver.cc:460] Webserver started at http://127.0.98.2:43915/ using document root <none> and password file <none>
I20260929 17:57:03.870113 20327 fs_manager.cc:359] Metadata directory not provided
I20260929 17:57:03.870179 20327 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:57:03.870385 20327 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:57:03.872486 20327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790704375975685-392-0/minicluster-data/ts-1/data/instance:
uuid: "9e8b527e63764e00a5077ae3b4b669fd"
format_stamp: "Formatted at 2026-09-29 17:57:03 on dist-test-slave-53nh"
server_key: "e1b3fe170d8ff345cb2c58d88087a545"
server_key_iv: "87a46fdfb8b928ed8ada0620eabe82c2"
server_key_version: "encryptionkey@0"
I20260929 17:57:03.873075 20327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790704375975685-392-0/minicluster-data/ts-1/wal/instance:
uuid: "9e8b527e63764e00a5077ae3b4b669fd"
format_stamp: "Formatted at 2026-09-29 17:57:03 on dist-test-slave-53nh"
server_key: "e1b3fe170d8ff345cb2c58d88087a545"
server_key_iv: "87a46fdfb8b928ed8ada0620eabe82c2"
server_key_version: "encryptionkey@0"
I20260929 17:57:03.876734 20327 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260929 17:57:03.879194 20341 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:57:03.880488 20327 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 17:57:03.880604 20327 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790704375975685-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790704375975685-392-0/minicluster-data/ts-1/wal
uuid: "9e8b527e63764e00a5077ae3b4b669fd"
format_stamp: "Formatted at 2026-09-29 17:57:03 on dist-test-slave-53nh"
server_key: "e1b3fe170d8ff345cb2c58d88087a545"
server_key_iv: "87a46fdfb8b928ed8ada0620eabe82c2"
server_key_version: "encryptionkey@0"
I20260929 17:57:03.880709 20327 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790704375975685-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790704375975685-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790704375975685-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:57:03.916332 20327 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:57:03.917093 20327 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:57:03.917263 20327 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:57:03.917865 20327 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 17:57:03.918820 20327 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:57:03.918867 20327 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:57:03.918919 20327 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:57:03.918947 20327 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 17:57:03.927220 20327 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36629
I20260929 17:57:03.927289 20454 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36629 every 8 connection(s)
I20260929 17:57:03.928164 20327 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790704375975685-392-0/minicluster-data/ts-1/data/info.pb
I20260929 17:57:03.936961 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskup3Soe/build/debug/bin/kudu as pid 20327
I20260929 17:57:03.937098 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790704375975685-392-0/minicluster-data/ts-1/wal/instance
I20260929 17:57:03.937382 392 external_mini_cluster.cc:1428] Setting key cb99d43d27a5d96fe10672f2aaad8f6f
I20260929 17:57:03.937700 20175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:03.930217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55595 (local address 127.0.98.62:36629)
0929 17:57:03.930409 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0929 17:57:03.930412 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 17:57:03.931526 (+ 1114us) server_negotiation.cc:407] Connection header received
0929 17:57:03.932913 (+ 1387us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:57:03.932916 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:57:03.932965 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:57:03.933074 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:57:03.934027 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:03.934588 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:03.935754 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:03.936026 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:03.936103 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:03.937211 (+ 1108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:57:03.937214 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:57:03.937216 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 17:57:03.937229 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:57:03.937254 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 17:57:03.937264 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:57:03.937309 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0929 17:57:03.937497 (+ 188us) server_negotiation.cc:299] Negotiation successful
0929 17:57:03.937556 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260929 17:57:03.938800 20457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:03.930650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36629 (local address 127.0.98.2:55595)
0929 17:57:03.931373 (+ 723us) negotiation.cc:107] Waiting for socket to connect
0929 17:57:03.931417 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 17:57:03.932701 (+ 1284us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:57:03.933269 (+ 568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:57:03.933277 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:57:03.933368 (+ 91us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:57:03.933886 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:57:03.933900 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:57:03.934686 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:57:03.934689 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:57:03.935664 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:57:03.935674 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:57:03.936155 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:57:03.936158 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:57:03.936305 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:03.936904 (+ 599us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 17:57:03.936905 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:57:03.936947 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 17:57:03.937039 (+ 92us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:57:03.937369 (+ 330us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:57:03.937376 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:57:03.937380 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 17:57:03.937484 (+ 104us) client_negotiation.cc:240] Negotiation successful
0929 17:57:03.937782 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":126,"threads_started":1}
I20260929 17:57:03.939445 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskup3Soe/build/debug/bin/kudu
/tmp/dist-test-taskup3Soe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790704375975685-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790704375975685-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790704375975685-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790704375975685-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36629
--builtin_ntp_servers=127.0.98.20:39903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 17:57:03.939935 20455 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36629
I20260929 17:57:03.940179 20455 heartbeater.cc:464] Registering TS with master...
I20260929 17:57:03.940764 20455 heartbeater.cc:511] Master 127.0.98.62:36629 requested a full tablet report, sending...
I20260929 17:57:03.941826 20136 ts_manager.cc:194] Registered new tserver with Master: 9e8b527e63764e00a5077ae3b4b669fd (127.0.98.2:36629)
I20260929 17:57:03.942370 20136 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:55595
W20260929 17:57:04.049038 20458 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:57:04.049333 20458 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:57:04.049387 20458 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:57:04.053489 20458 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:57:04.053555 20458 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:57:04.053630 20458 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 17:57:04.058511 20458 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790704375975685-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790704375975685-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790704375975685-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36629
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20458
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790704375975685-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:57:04.059612 20458 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:57:04.061026 20458 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:57:04.070596 20464 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:57:04.070631 20463 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:57:04.070600 20466 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:57:04.070891 20458 server_base.cc:1034] running on GCE node
I20260929 17:57:04.071363 20458 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:57:04.072012 20458 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:57:04.073169 20458 hybrid_clock.cc:648] HybridClock initialized: now 1790704624073137 us; error 41 us; skew 500 ppm
I20260929 17:57:04.078596 20458 webserver.cc:460] Webserver started at http://127.0.98.3:39261/ using document root <none> and password file <none>
I20260929 17:57:04.079190 20458 fs_manager.cc:359] Metadata directory not provided
I20260929 17:57:04.079252 20458 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:57:04.079420 20458 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:57:04.081519 20458 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790704375975685-392-0/minicluster-data/ts-2/data/instance:
uuid: "c6c146b64e564bfe9e12095fa5d9ebf9"
format_stamp: "Formatted at 2026-09-29 17:57:04 on dist-test-slave-53nh"
server_key: "034bc0579e7772cddea84fa85a92913f"
server_key_iv: "40bb67e13eeb209aa34e257954d21dce"
server_key_version: "encryptionkey@0"
I20260929 17:57:04.082016 20458 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790704375975685-392-0/minicluster-data/ts-2/wal/instance:
uuid: "c6c146b64e564bfe9e12095fa5d9ebf9"
format_stamp: "Formatted at 2026-09-29 17:57:04 on dist-test-slave-53nh"
server_key: "034bc0579e7772cddea84fa85a92913f"
server_key_iv: "40bb67e13eeb209aa34e257954d21dce"
server_key_version: "encryptionkey@0"
I20260929 17:57:04.085772 20458 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260929 17:57:04.088331 20472 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:57:04.089746 20458 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 17:57:04.089886 20458 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790704375975685-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790704375975685-392-0/minicluster-data/ts-2/wal
uuid: "c6c146b64e564bfe9e12095fa5d9ebf9"
format_stamp: "Formatted at 2026-09-29 17:57:04 on dist-test-slave-53nh"
server_key: "034bc0579e7772cddea84fa85a92913f"
server_key_iv: "40bb67e13eeb209aa34e257954d21dce"
server_key_version: "encryptionkey@0"
I20260929 17:57:04.089989 20458 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790704375975685-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790704375975685-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790704375975685-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:57:04.108312 20458 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:57:04.109036 20458 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:57:04.109198 20458 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:57:04.109815 20458 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 17:57:04.110776 20458 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 17:57:04.110823 20458 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 17:57:04.110870 20458 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 17:57:04.110884 20458 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 17:57:04.119225 20458 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41975
I20260929 17:57:04.119258 20585 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41975 every 8 connection(s)
I20260929 17:57:04.120182 20458 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790704375975685-392-0/minicluster-data/ts-2/data/info.pb
I20260929 17:57:04.125999 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskup3Soe/build/debug/bin/kudu as pid 20458
I20260929 17:57:04.126199 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790704375975685-392-0/minicluster-data/ts-2/wal/instance
I20260929 17:57:04.126516 392 external_mini_cluster.cc:1428] Setting key 2961ea7db45d58e7f482658270b8bb15
I20260929 17:57:04.130254 20175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:04.122094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:32933 (local address 127.0.98.62:36629)
0929 17:57:04.122249 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0929 17:57:04.122254 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:57:04.123379 (+ 1125us) server_negotiation.cc:407] Connection header received
0929 17:57:04.124708 (+ 1329us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:57:04.124711 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:57:04.124762 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:57:04.124856 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:57:04.126147 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:04.126783 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:04.128176 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:04.128496 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:04.128610 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:04.129767 (+ 1157us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:57:04.129772 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:57:04.129774 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 17:57:04.129787 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:57:04.129811 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 17:57:04.129819 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:57:04.129878 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0929 17:57:04.130037 (+ 159us) server_negotiation.cc:299] Negotiation successful
0929 17:57:04.130106 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 17:57:04.131161 20588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:04.122532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36629 (local address 127.0.98.3:32933)
0929 17:57:04.123218 (+ 686us) negotiation.cc:107] Waiting for socket to connect
0929 17:57:04.123258 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 17:57:04.124486 (+ 1228us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:57:04.125100 (+ 614us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:57:04.125108 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:57:04.125203 (+ 95us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:57:04.125943 (+ 740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:57:04.125969 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:57:04.126904 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:57:04.126908 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:57:04.128066 (+ 1158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:57:04.128077 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:57:04.128652 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:57:04.128657 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:57:04.128840 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:04.129484 (+ 644us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 17:57:04.129485 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:57:04.129531 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 17:57:04.129602 (+ 71us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:57:04.129889 (+ 287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:57:04.129899 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:57:04.129906 (+ 7us) client_negotiation.cc:769] Sending connection context
0929 17:57:04.130026 (+ 120us) client_negotiation.cc:240] Negotiation successful
0929 17:57:04.130348 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":120,"threads_started":1}
I20260929 17:57:04.132364 20586 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36629
I20260929 17:57:04.132622 20586 heartbeater.cc:464] Registering TS with master...
I20260929 17:57:04.133229 20586 heartbeater.cc:511] Master 127.0.98.62:36629 requested a full tablet report, sending...
I20260929 17:57:04.134280 20136 ts_manager.cc:194] Registered new tserver with Master: c6c146b64e564bfe9e12095fa5d9ebf9 (127.0.98.3:41975)
I20260929 17:57:04.134871 20136 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:32933
I20260929 17:57:04.140810 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 17:57:04.146375 20175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:04.142734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60160 (local address 127.0.98.62:36629)
0929 17:57:04.142868 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0929 17:57:04.142872 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:57:04.143037 (+ 165us) server_negotiation.cc:407] Connection header received
0929 17:57:04.143171 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:57:04.143174 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:57:04.143214 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:57:04.143273 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:57:04.143550 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:04.144071 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:04.144876 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:04.145126 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:04.145207 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:04.145986 (+ 779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:57:04.145990 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:57:04.145993 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 17:57:04.146005 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:57:04.146026 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 17:57:04.146034 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:57:04.146090 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0929 17:57:04.146171 (+ 81us) server_negotiation.cc:299] Negotiation successful
0929 17:57:04.146227 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 17:57:04.149295 20136 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60160:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 17:57:04.152249 20136 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 17:57:04.165488 20597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:04.160553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36629 (local address 127.0.0.1:56456)
0929 17:57:04.160898 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0929 17:57:04.160951 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0929 17:57:04.161117 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:57:04.161639 (+ 522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:57:04.161646 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:57:04.161672 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:57:04.161847 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:57:04.161857 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:57:04.163043 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:57:04.163046 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:57:04.163908 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:57:04.163917 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:57:04.164948 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:57:04.164951 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:57:04.165082 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:04.165128 (+ 46us) client_negotiation.cc:769] Sending connection context
0929 17:57:04.165217 (+ 89us) client_negotiation.cc:240] Negotiation successful
0929 17:57:04.165336 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":74,"threads_started":1}
I20260929 17:57:04.166401 20598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:04.160686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56456 (local address 127.0.98.2:36629)
0929 17:57:04.161135 (+ 449us) server_negotiation.cc:206] Beginning negotiation
0929 17:57:04.161147 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 17:57:04.161173 (+ 26us) server_negotiation.cc:407] Connection header received
0929 17:57:04.161261 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:57:04.161266 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:57:04.161472 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:57:04.161632 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:57:04.161977 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:04.162954 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:04.164030 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:04.164825 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:04.164934 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:04.166085 (+ 1151us) server_negotiation.cc:1035] Waiting for connection context
0929 17:57:04.166139 (+ 54us) server_negotiation.cc:299] Negotiation successful
0929 17:57:04.166238 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":356,"thread_start_us":114,"threads_started":1}
I20260929 17:57:04.167281 20601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:04.161295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41975 (local address 127.0.0.1:55386)
0929 17:57:04.161857 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0929 17:57:04.161868 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 17:57:04.161930 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:57:04.162554 (+ 624us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:57:04.162557 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:57:04.162575 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:57:04.162658 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:57:04.162663 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:57:04.164066 (+ 1403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:57:04.164069 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:57:04.165635 (+ 1566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:57:04.165647 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:57:04.166916 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:57:04.166919 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:57:04.166999 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:04.167020 (+ 21us) client_negotiation.cc:769] Sending connection context
0929 17:57:04.167078 (+ 58us) client_negotiation.cc:240] Negotiation successful
0929 17:57:04.167144 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":508,"thread_start_us":76,"threads_started":1}
I20260929 17:57:04.167312 20599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:04.160774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53136 (local address 127.0.98.1:42817)
0929 17:57:04.161412 (+ 638us) server_negotiation.cc:206] Beginning negotiation
0929 17:57:04.161423 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 17:57:04.161786 (+ 363us) server_negotiation.cc:407] Connection header received
0929 17:57:04.162000 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:57:04.162005 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:57:04.162195 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:57:04.162310 (+ 115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:57:04.162748 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:04.163919 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:04.165089 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:04.165849 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:04.165945 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:04.166103 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0929 17:57:04.166437 (+ 334us) server_negotiation.cc:299] Negotiation successful
0929 17:57:04.166544 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":542,"thread_start_us":123,"threads_started":1}
I20260929 17:57:04.167372 20602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:04.161486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55386 (local address 127.0.98.3:41975)
0929 17:57:04.162149 (+ 663us) server_negotiation.cc:206] Beginning negotiation
0929 17:57:04.162159 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 17:57:04.162180 (+ 21us) server_negotiation.cc:407] Connection header received
0929 17:57:04.162259 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:57:04.162264 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:57:04.162424 (+ 160us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:57:04.162557 (+ 133us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:57:04.162781 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:04.163922 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:04.165779 (+ 1857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:04.166803 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:04.166904 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:04.167084 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0929 17:57:04.167132 (+ 48us) server_negotiation.cc:299] Negotiation successful
0929 17:57:04.167217 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":572,"thread_start_us":136,"threads_started":1}
I20260929 17:57:04.167695 20600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:04.160555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42817 (local address 127.0.0.1:53136)
0929 17:57:04.161681 (+ 1126us) negotiation.cc:107] Waiting for socket to connect
0929 17:57:04.161693 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 17:57:04.161852 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:57:04.162473 (+ 621us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:57:04.162476 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:57:04.162491 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:57:04.162604 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:57:04.162611 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:57:04.164046 (+ 1435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:57:04.164049 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:57:04.164937 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:57:04.164945 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:57:04.166159 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:57:04.166163 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:57:04.166268 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:04.166290 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 17:57:04.167498 (+ 1208us) client_negotiation.cc:240] Negotiation successful
0929 17:57:04.167574 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1065,"mutex_wait_us":505,"thread_start_us":94,"threads_started":1}
I20260929 17:57:04.169477 20389 tablet_service.cc:1467] Processing CreateTablet for tablet 743383a80b0f470193e2155d6f1a860d (DEFAULT_TABLE table=test-table [id=0f7dd12dcf9d4cc6822f835fa9cb9a00]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 17:57:04.170488 20389 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 743383a80b0f470193e2155d6f1a860d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:57:04.170233 20258 tablet_service.cc:1467] Processing CreateTablet for tablet 743383a80b0f470193e2155d6f1a860d (DEFAULT_TABLE table=test-table [id=0f7dd12dcf9d4cc6822f835fa9cb9a00]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 17:57:04.170233 20520 tablet_service.cc:1467] Processing CreateTablet for tablet 743383a80b0f470193e2155d6f1a860d (DEFAULT_TABLE table=test-table [id=0f7dd12dcf9d4cc6822f835fa9cb9a00]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 17:57:04.171254 20258 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 743383a80b0f470193e2155d6f1a860d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:57:04.171254 20520 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 743383a80b0f470193e2155d6f1a860d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:57:04.177575 20603 tablet_bootstrap.cc:492] T 743383a80b0f470193e2155d6f1a860d P 9e8b527e63764e00a5077ae3b4b669fd: Bootstrap starting.
I20260929 17:57:04.178227 20605 tablet_bootstrap.cc:492] T 743383a80b0f470193e2155d6f1a860d P c5218c5d07194509bc7c6715cb028c11: Bootstrap starting.
I20260929 17:57:04.180016 20603 tablet_bootstrap.cc:654] T 743383a80b0f470193e2155d6f1a860d P 9e8b527e63764e00a5077ae3b4b669fd: Neither blocks nor log segments found. Creating new log.
I20260929 17:57:04.180306 20605 tablet_bootstrap.cc:654] T 743383a80b0f470193e2155d6f1a860d P c5218c5d07194509bc7c6715cb028c11: Neither blocks nor log segments found. Creating new log.
I20260929 17:57:04.180900 20603 log.cc:824] T 743383a80b0f470193e2155d6f1a860d P 9e8b527e63764e00a5077ae3b4b669fd: Log is configured to *not* fsync() on all Append() calls
I20260929 17:57:04.181048 20605 log.cc:824] T 743383a80b0f470193e2155d6f1a860d P c5218c5d07194509bc7c6715cb028c11: Log is configured to *not* fsync() on all Append() calls
I20260929 17:57:04.181650 20604 tablet_bootstrap.cc:492] T 743383a80b0f470193e2155d6f1a860d P c6c146b64e564bfe9e12095fa5d9ebf9: Bootstrap starting.
I20260929 17:57:04.183130 20603 tablet_bootstrap.cc:492] T 743383a80b0f470193e2155d6f1a860d P 9e8b527e63764e00a5077ae3b4b669fd: No bootstrap required, opened a new log
I20260929 17:57:04.183233 20605 tablet_bootstrap.cc:492] T 743383a80b0f470193e2155d6f1a860d P c5218c5d07194509bc7c6715cb028c11: No bootstrap required, opened a new log
I20260929 17:57:04.183393 20603 ts_tablet_manager.cc:1397] T 743383a80b0f470193e2155d6f1a860d P 9e8b527e63764e00a5077ae3b4b669fd: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260929 17:57:04.183434 20605 ts_tablet_manager.cc:1397] T 743383a80b0f470193e2155d6f1a860d P c5218c5d07194509bc7c6715cb028c11: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260929 17:57:04.183853 20604 tablet_bootstrap.cc:654] T 743383a80b0f470193e2155d6f1a860d P c6c146b64e564bfe9e12095fa5d9ebf9: Neither blocks nor log segments found. Creating new log.
I20260929 17:57:04.184664 20604 log.cc:824] T 743383a80b0f470193e2155d6f1a860d P c6c146b64e564bfe9e12095fa5d9ebf9: Log is configured to *not* fsync() on all Append() calls
I20260929 17:57:04.186817 20604 tablet_bootstrap.cc:492] T 743383a80b0f470193e2155d6f1a860d P c6c146b64e564bfe9e12095fa5d9ebf9: No bootstrap required, opened a new log
I20260929 17:57:04.187023 20604 ts_tablet_manager.cc:1397] T 743383a80b0f470193e2155d6f1a860d P c6c146b64e564bfe9e12095fa5d9ebf9: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260929 17:57:04.187222 20605 raft_consensus.cc:348] T 743383a80b0f470193e2155d6f1a860d P c5218c5d07194509bc7c6715cb028c11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e8b527e63764e00a5077ae3b4b669fd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36629 } } peers { permanent_uuid: "c5218c5d07194509bc7c6715cb028c11" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42817 } } peers { permanent_uuid: "c6c146b64e564bfe9e12095fa5d9ebf9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41975 } }
I20260929 17:57:04.187466 20605 raft_consensus.cc:374] T 743383a80b0f470193e2155d6f1a860d P c5218c5d07194509bc7c6715cb028c11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:57:04.187520 20605 raft_consensus.cc:729] T 743383a80b0f470193e2155d6f1a860d P c5218c5d07194509bc7c6715cb028c11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5218c5d07194509bc7c6715cb028c11, State: Initialized, Role: FOLLOWER
I20260929 17:57:04.187492 20603 raft_consensus.cc:348] T 743383a80b0f470193e2155d6f1a860d P 9e8b527e63764e00a5077ae3b4b669fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e8b527e63764e00a5077ae3b4b669fd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36629 } } peers { permanent_uuid: "c5218c5d07194509bc7c6715cb028c11" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42817 } } peers { permanent_uuid: "c6c146b64e564bfe9e12095fa5d9ebf9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41975 } }
I20260929 17:57:04.187749 20603 raft_consensus.cc:374] T 743383a80b0f470193e2155d6f1a860d P 9e8b527e63764e00a5077ae3b4b669fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:57:04.187798 20603 raft_consensus.cc:729] T 743383a80b0f470193e2155d6f1a860d P 9e8b527e63764e00a5077ae3b4b669fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e8b527e63764e00a5077ae3b4b669fd, State: Initialized, Role: FOLLOWER
I20260929 17:57:04.188000 20605 consensus_queue.cc:260] T 743383a80b0f470193e2155d6f1a860d P c5218c5d07194509bc7c6715cb028c11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e8b527e63764e00a5077ae3b4b669fd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36629 } } peers { permanent_uuid: "c5218c5d07194509bc7c6715cb028c11" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42817 } } peers { permanent_uuid: "c6c146b64e564bfe9e12095fa5d9ebf9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41975 } }
I20260929 17:57:04.188195 20603 consensus_queue.cc:260] T 743383a80b0f470193e2155d6f1a860d P 9e8b527e63764e00a5077ae3b4b669fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e8b527e63764e00a5077ae3b4b669fd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36629 } } peers { permanent_uuid: "c5218c5d07194509bc7c6715cb028c11" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42817 } } peers { permanent_uuid: "c6c146b64e564bfe9e12095fa5d9ebf9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41975 } }
I20260929 17:57:04.188735 20324 heartbeater.cc:503] Master 127.0.98.62:36629 was elected leader, sending a full tablet report...
I20260929 17:57:04.189100 20605 ts_tablet_manager.cc:1428] T 743383a80b0f470193e2155d6f1a860d P c5218c5d07194509bc7c6715cb028c11: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260929 17:57:04.189131 20603 ts_tablet_manager.cc:1428] T 743383a80b0f470193e2155d6f1a860d P 9e8b527e63764e00a5077ae3b4b669fd: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260929 17:57:04.189502 20455 heartbeater.cc:503] Master 127.0.98.62:36629 was elected leader, sending a full tablet report...
I20260929 17:57:04.191476 20604 raft_consensus.cc:348] T 743383a80b0f470193e2155d6f1a860d P c6c146b64e564bfe9e12095fa5d9ebf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e8b527e63764e00a5077ae3b4b669fd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36629 } } peers { permanent_uuid: "c5218c5d07194509bc7c6715cb028c11" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42817 } } peers { permanent_uuid: "c6c146b64e564bfe9e12095fa5d9ebf9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41975 } }
I20260929 17:57:04.191742 20604 raft_consensus.cc:374] T 743383a80b0f470193e2155d6f1a860d P c6c146b64e564bfe9e12095fa5d9ebf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:57:04.191817 20604 raft_consensus.cc:729] T 743383a80b0f470193e2155d6f1a860d P c6c146b64e564bfe9e12095fa5d9ebf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6c146b64e564bfe9e12095fa5d9ebf9, State: Initialized, Role: FOLLOWER
I20260929 17:57:04.192248 20604 consensus_queue.cc:260] T 743383a80b0f470193e2155d6f1a860d P c6c146b64e564bfe9e12095fa5d9ebf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e8b527e63764e00a5077ae3b4b669fd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36629 } } peers { permanent_uuid: "c5218c5d07194509bc7c6715cb028c11" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42817 } } peers { permanent_uuid: "c6c146b64e564bfe9e12095fa5d9ebf9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41975 } }
I20260929 17:57:04.192828 20586 heartbeater.cc:503] Master 127.0.98.62:36629 was elected leader, sending a full tablet report...
I20260929 17:57:04.193329 20604 ts_tablet_manager.cc:1428] T 743383a80b0f470193e2155d6f1a860d P c6c146b64e564bfe9e12095fa5d9ebf9: Time spent starting tablet: real 0.006s user 0.006s sys 0.002s
W20260929 17:57:04.227077 20325 tablet.cc:2369] T 743383a80b0f470193e2155d6f1a860d P c5218c5d07194509bc7c6715cb028c11: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 17:57:04.228003 20611 raft_consensus.cc:482] T 743383a80b0f470193e2155d6f1a860d P c6c146b64e564bfe9e12095fa5d9ebf9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 17:57:04.228242 20611 raft_consensus.cc:504] T 743383a80b0f470193e2155d6f1a860d P c6c146b64e564bfe9e12095fa5d9ebf9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e8b527e63764e00a5077ae3b4b669fd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36629 } } peers { permanent_uuid: "c5218c5d07194509bc7c6715cb028c11" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42817 } } peers { permanent_uuid: "c6c146b64e564bfe9e12095fa5d9ebf9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41975 } }
I20260929 17:57:04.229485 20611 leader_election.cc:288] T 743383a80b0f470193e2155d6f1a860d P c6c146b64e564bfe9e12095fa5d9ebf9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9e8b527e63764e00a5077ae3b4b669fd (127.0.98.2:36629), c5218c5d07194509bc7c6715cb028c11 (127.0.98.1:42817)
I20260929 17:57:04.234591 20599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:04.229974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44487 (local address 127.0.98.1:42817)
0929 17:57:04.230112 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0929 17:57:04.230116 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:57:04.230308 (+ 192us) server_negotiation.cc:407] Connection header received
0929 17:57:04.230416 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:57:04.230420 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:57:04.230463 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:57:04.230520 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:57:04.230802 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:04.231510 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:04.233162 (+ 1652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:04.233807 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:04.233935 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:04.234005 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0929 17:57:04.234351 (+ 346us) server_negotiation.cc:299] Negotiation successful
0929 17:57:04.234442 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 17:57:04.234602 20598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:04.229852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43653 (local address 127.0.98.2:36629)
0929 17:57:04.229996 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0929 17:57:04.230000 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:57:04.230136 (+ 136us) server_negotiation.cc:407] Connection header received
0929 17:57:04.230198 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:57:04.230201 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:57:04.230245 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:57:04.230325 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 17:57:04.230621 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:04.231433 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:04.233162 (+ 1729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:04.233851 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:04.233965 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:04.234015 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0929 17:57:04.234355 (+ 340us) server_negotiation.cc:299] Negotiation successful
0929 17:57:04.234438 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 17:57:04.234604 20588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:04.229741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36629 (local address 127.0.98.3:43653)
0929 17:57:04.229975 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0929 17:57:04.229991 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 17:57:04.230108 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:57:04.230347 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:57:04.230350 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:57:04.230365 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:57:04.230499 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:57:04.230507 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:57:04.231578 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:57:04.231583 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:57:04.232970 (+ 1387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:57:04.232986 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:57:04.233977 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:57:04.233982 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:57:04.234138 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:04.234213 (+ 75us) client_negotiation.cc:769] Sending connection context
0929 17:57:04.234304 (+ 91us) client_negotiation.cc:240] Negotiation successful
0929 17:57:04.234419 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"mutex_wait_us":24}
I20260929 17:57:04.234633 20612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:04.229844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42817 (local address 127.0.98.3:44487)
0929 17:57:04.230217 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0929 17:57:04.230231 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 17:57:04.230323 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:57:04.230553 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:57:04.230557 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:57:04.230575 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 17:57:04.230699 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:57:04.230706 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:57:04.231632 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:57:04.231635 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:57:04.232970 (+ 1335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:57:04.232986 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:57:04.233977 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:57:04.233982 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:57:04.234202 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:04.234224 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 17:57:04.234304 (+ 80us) client_negotiation.cc:240] Negotiation successful
0929 17:57:04.234419 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306,"mutex_wait_us":190,"thread_start_us":124,"threads_started":1}
I20260929 17:57:04.235412 20409 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "743383a80b0f470193e2155d6f1a860d" candidate_uuid: "c6c146b64e564bfe9e12095fa5d9ebf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e8b527e63764e00a5077ae3b4b669fd" is_pre_election: true
I20260929 17:57:04.235448 20278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "743383a80b0f470193e2155d6f1a860d" candidate_uuid: "c6c146b64e564bfe9e12095fa5d9ebf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5218c5d07194509bc7c6715cb028c11" is_pre_election: true
I20260929 17:57:04.235730 20409 raft_consensus.cc:2454] T 743383a80b0f470193e2155d6f1a860d P 9e8b527e63764e00a5077ae3b4b669fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6c146b64e564bfe9e12095fa5d9ebf9 in term 0.
I20260929 17:57:04.235730 20278 raft_consensus.cc:2454] T 743383a80b0f470193e2155d6f1a860d P c5218c5d07194509bc7c6715cb028c11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6c146b64e564bfe9e12095fa5d9ebf9 in term 0.
I20260929 17:57:04.236244 20473 leader_election.cc:302] T 743383a80b0f470193e2155d6f1a860d P c6c146b64e564bfe9e12095fa5d9ebf9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9e8b527e63764e00a5077ae3b4b669fd, c6c146b64e564bfe9e12095fa5d9ebf9; no voters:
I20260929 17:57:04.236536 20611 raft_consensus.cc:2781] T 743383a80b0f470193e2155d6f1a860d P c6c146b64e564bfe9e12095fa5d9ebf9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 17:57:04.236618 20611 raft_consensus.cc:482] T 743383a80b0f470193e2155d6f1a860d P c6c146b64e564bfe9e12095fa5d9ebf9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 17:57:04.236668 20611 raft_consensus.cc:3037] T 743383a80b0f470193e2155d6f1a860d P c6c146b64e564bfe9e12095fa5d9ebf9 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:57:04.237743 20611 raft_consensus.cc:504] T 743383a80b0f470193e2155d6f1a860d P c6c146b64e564bfe9e12095fa5d9ebf9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e8b527e63764e00a5077ae3b4b669fd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36629 } } peers { permanent_uuid: "c5218c5d07194509bc7c6715cb028c11" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42817 } } peers { permanent_uuid: "c6c146b64e564bfe9e12095fa5d9ebf9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41975 } }
I20260929 17:57:04.238075 20611 leader_election.cc:288] T 743383a80b0f470193e2155d6f1a860d P c6c146b64e564bfe9e12095fa5d9ebf9 [CANDIDATE]: Term 1 election: Requested vote from peers 9e8b527e63764e00a5077ae3b4b669fd (127.0.98.2:36629), c5218c5d07194509bc7c6715cb028c11 (127.0.98.1:42817)
I20260929 17:57:04.238432 20409 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "743383a80b0f470193e2155d6f1a860d" candidate_uuid: "c6c146b64e564bfe9e12095fa5d9ebf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e8b527e63764e00a5077ae3b4b669fd"
I20260929 17:57:04.238449 20278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "743383a80b0f470193e2155d6f1a860d" candidate_uuid: "c6c146b64e564bfe9e12095fa5d9ebf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5218c5d07194509bc7c6715cb028c11"
I20260929 17:57:04.238562 20409 raft_consensus.cc:3037] T 743383a80b0f470193e2155d6f1a860d P 9e8b527e63764e00a5077ae3b4b669fd [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:57:04.238600 20278 raft_consensus.cc:3037] T 743383a80b0f470193e2155d6f1a860d P c5218c5d07194509bc7c6715cb028c11 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:57:04.239583 20409 raft_consensus.cc:2454] T 743383a80b0f470193e2155d6f1a860d P 9e8b527e63764e00a5077ae3b4b669fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6c146b64e564bfe9e12095fa5d9ebf9 in term 1.
I20260929 17:57:04.239632 20278 raft_consensus.cc:2454] T 743383a80b0f470193e2155d6f1a860d P c5218c5d07194509bc7c6715cb028c11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6c146b64e564bfe9e12095fa5d9ebf9 in term 1.
I20260929 17:57:04.239867 20473 leader_election.cc:302] T 743383a80b0f470193e2155d6f1a860d P c6c146b64e564bfe9e12095fa5d9ebf9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9e8b527e63764e00a5077ae3b4b669fd, c6c146b64e564bfe9e12095fa5d9ebf9; no voters:
I20260929 17:57:04.240041 20611 raft_consensus.cc:2781] T 743383a80b0f470193e2155d6f1a860d P c6c146b64e564bfe9e12095fa5d9ebf9 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 17:57:04.240275 20611 raft_consensus.cc:686] T 743383a80b0f470193e2155d6f1a860d P c6c146b64e564bfe9e12095fa5d9ebf9 [term 1 LEADER]: Becoming Leader. State: Replica: c6c146b64e564bfe9e12095fa5d9ebf9, State: Running, Role: LEADER
I20260929 17:57:04.240615 20611 consensus_queue.cc:237] T 743383a80b0f470193e2155d6f1a860d P c6c146b64e564bfe9e12095fa5d9ebf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e8b527e63764e00a5077ae3b4b669fd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36629 } } peers { permanent_uuid: "c5218c5d07194509bc7c6715cb028c11" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42817 } } peers { permanent_uuid: "c6c146b64e564bfe9e12095fa5d9ebf9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41975 } }
I20260929 17:57:04.243911 20133 catalog_manager.cc:5495] T 743383a80b0f470193e2155d6f1a860d P c6c146b64e564bfe9e12095fa5d9ebf9 reported cstate change: term changed from 0 to 1, leader changed from <none> to c6c146b64e564bfe9e12095fa5d9ebf9 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "c6c146b64e564bfe9e12095fa5d9ebf9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e8b527e63764e00a5077ae3b4b669fd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5218c5d07194509bc7c6715cb028c11" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c6c146b64e564bfe9e12095fa5d9ebf9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41975 } health_report { overall_health: HEALTHY } } }
I20260929 17:57:04.266192 20602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:04.263016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55392 (local address 127.0.98.3:41975)
0929 17:57:04.263139 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0929 17:57:04.263143 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:57:04.263156 (+ 13us) server_negotiation.cc:407] Connection header received
0929 17:57:04.263253 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:57:04.263255 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:57:04.263307 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:57:04.263399 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 17:57:04.263573 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:04.264069 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:04.264872 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:04.265091 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:04.265155 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:04.265413 (+ 258us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 17:57:04.265828 (+ 415us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 17:57:04.265916 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0929 17:57:04.265985 (+ 69us) server_negotiation.cc:299] Negotiation successful
0929 17:57:04.266030 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 17:57:04.272028 20409 raft_consensus.cc:1261] T 743383a80b0f470193e2155d6f1a860d P 9e8b527e63764e00a5077ae3b4b669fd [term 1 FOLLOWER]: Refusing update from remote peer c6c146b64e564bfe9e12095fa5d9ebf9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 17:57:04.272029 20278 raft_consensus.cc:1261] T 743383a80b0f470193e2155d6f1a860d P c5218c5d07194509bc7c6715cb028c11 [term 1 FOLLOWER]: Refusing update from remote peer c6c146b64e564bfe9e12095fa5d9ebf9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 17:57:04.272713 20613 consensus_queue.cc:1035] T 743383a80b0f470193e2155d6f1a860d P c6c146b64e564bfe9e12095fa5d9ebf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e8b527e63764e00a5077ae3b4b669fd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 17:57:04.272905 20611 consensus_queue.cc:1035] T 743383a80b0f470193e2155d6f1a860d P c6c146b64e564bfe9e12095fa5d9ebf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5218c5d07194509bc7c6715cb028c11" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 17:57:04.289021 20133 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:60160:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 17:57:04.289225 20133 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:60160:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 17:57:04.291772 20133 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 02683cfb36704a498d3a87fa622236e4: Sending DeleteTablet for 3 replicas of tablet 743383a80b0f470193e2155d6f1a860d
I20260929 17:57:04.292707 20520 tablet_service.cc:1514] Processing DeleteTablet for tablet 743383a80b0f470193e2155d6f1a860d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 17:57:04 UTC) from {username='slave'} at 127.0.0.1:55386
I20260929 17:57:04.292716 20258 tablet_service.cc:1514] Processing DeleteTablet for tablet 743383a80b0f470193e2155d6f1a860d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 17:57:04 UTC) from {username='slave'} at 127.0.0.1:53136
I20260929 17:57:04.293319 20626 tablet_replica.cc:330] T 743383a80b0f470193e2155d6f1a860d P c6c146b64e564bfe9e12095fa5d9ebf9: stopping tablet replica
I20260929 17:57:04.293350 20389 tablet_service.cc:1514] Processing DeleteTablet for tablet 743383a80b0f470193e2155d6f1a860d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 17:57:04 UTC) from {username='slave'} at 127.0.0.1:56456
I20260929 17:57:04.293905 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskup3Soe/build/debug/bin/kudu with pid 20196
I20260929 17:57:04.293942 20626 raft_consensus.cc:2229] T 743383a80b0f470193e2155d6f1a860d P c6c146b64e564bfe9e12095fa5d9ebf9 [term 1 LEADER]: Raft consensus shutting down.
I20260929 17:57:04.295104 20626 raft_consensus.cc:2258] T 743383a80b0f470193e2155d6f1a860d P c6c146b64e564bfe9e12095fa5d9ebf9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 17:57:04.295661 20628 tablet_replica.cc:330] T 743383a80b0f470193e2155d6f1a860d P 9e8b527e63764e00a5077ae3b4b669fd: stopping tablet replica
I20260929 17:57:04.296022 20628 raft_consensus.cc:2229] T 743383a80b0f470193e2155d6f1a860d P 9e8b527e63764e00a5077ae3b4b669fd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 17:57:04.296370 20628 raft_consensus.cc:2258] T 743383a80b0f470193e2155d6f1a860d P 9e8b527e63764e00a5077ae3b4b669fd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 17:57:04.296797 20626 ts_tablet_manager.cc:1905] T 743383a80b0f470193e2155d6f1a860d P c6c146b64e564bfe9e12095fa5d9ebf9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 17:57:04.297652 20628 ts_tablet_manager.cc:1905] T 743383a80b0f470193e2155d6f1a860d P 9e8b527e63764e00a5077ae3b4b669fd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 17:57:04.299526 20626 ts_tablet_manager.cc:1918] T 743383a80b0f470193e2155d6f1a860d P c6c146b64e564bfe9e12095fa5d9ebf9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 17:57:04.299624 20626 log.cc:1196] T 743383a80b0f470193e2155d6f1a860d P c6c146b64e564bfe9e12095fa5d9ebf9: Deleting WAL directory at /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790704375975685-392-0/minicluster-data/ts-2/wal/wals/743383a80b0f470193e2155d6f1a860d
I20260929 17:57:04.299654 20628 ts_tablet_manager.cc:1918] T 743383a80b0f470193e2155d6f1a860d P 9e8b527e63764e00a5077ae3b4b669fd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 17:57:04.299739 20628 log.cc:1196] T 743383a80b0f470193e2155d6f1a860d P 9e8b527e63764e00a5077ae3b4b669fd: Deleting WAL directory at /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790704375975685-392-0/minicluster-data/ts-1/wal/wals/743383a80b0f470193e2155d6f1a860d
I20260929 17:57:04.299932 20626 ts_tablet_manager.cc:1939] T 743383a80b0f470193e2155d6f1a860d P c6c146b64e564bfe9e12095fa5d9ebf9: Deleting consensus metadata
I20260929 17:57:04.300004 20628 ts_tablet_manager.cc:1939] T 743383a80b0f470193e2155d6f1a860d P 9e8b527e63764e00a5077ae3b4b669fd: Deleting consensus metadata
I20260929 17:57:04.301100 20120 catalog_manager.cc:4841] TS 9e8b527e63764e00a5077ae3b4b669fd (127.0.98.2:36629): tablet 743383a80b0f470193e2155d6f1a860d (table test-table [id=0f7dd12dcf9d4cc6822f835fa9cb9a00]) successfully deleted
I20260929 17:57:04.305449 20120 catalog_manager.cc:4841] TS c6c146b64e564bfe9e12095fa5d9ebf9 (127.0.98.3:41975): tablet 743383a80b0f470193e2155d6f1a860d (table test-table [id=0f7dd12dcf9d4cc6822f835fa9cb9a00]) successfully deleted
W20260929 17:57:04.305697 20121 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:42817 (error 108)
I20260929 17:57:04.306515 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskup3Soe/build/debug/bin/kudu with pid 20327
I20260929 17:57:04.306722 20600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:04.306414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42817 (local address 127.0.0.1:53146)
0929 17:57:04.306564 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0929 17:57:04.306635 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260929 17:57:04.307060 20121 catalog_manager.cc:4584] TS c5218c5d07194509bc7c6715cb028c11 (127.0.98.1:42817): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 743383a80b0f470193e2155d6f1a860d: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42817: connect: Connection refused (error 111)
I20260929 17:57:04.311604 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskup3Soe/build/debug/bin/kudu with pid 20458
I20260929 17:57:04.316143 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskup3Soe/build/debug/bin/kudu with pid 20105
2026-09-29T17:57:04Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (1063 ms)
[----------] 6 tests from EncryptionPolicyTest (9899 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790704375975685-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 17:57:04 dist-test-slave-53nh krb5kdc[20632](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 17:57:04 dist-test-slave-53nh krb5kdc[20632](info): set up 2 sockets
Sep 29 17:57:04 dist-test-slave-53nh krb5kdc[20632](info): commencing operation
krb5kdc: starting...
W20260929 17:57:05.285025 392 mini_kdc.cc:121] Time spent starting KDC: real 0.949s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790704375975685-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790704375975685-392-0/krb5kdc/test-user.keytab.
Sep 29 17:57:05 dist-test-slave-53nh krb5kdc[20632](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704625, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T17:57:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T17:57:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790704375975685-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790704375975685-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 17:57:05.403669 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskup3Soe/build/debug/bin/kudu
/tmp/dist-test-taskup3Soe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790704375975685-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790704375975685-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790704375975685-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790704375975685-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34143
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790704375975685-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34143
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790704375975685-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790704375975685-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790704375975685-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:57:05.516052 20645 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:57:05.516340 20645 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:57:05.516383 20645 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:57:05.520382 20645 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 17:57:05.520448 20645 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:57:05.520466 20645 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:57:05.520483 20645 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 17:57:05.520500 20645 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 17:57:05.525844 20645 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790704375975685-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790704375975685-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34143
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790704375975685-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34143
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790704375975685-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20645
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790704375975685-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:57:05.526918 20645 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:57:05.527904 20645 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:57:05.536653 20651 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:57:05.536831 20650 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:57:05.536672 20653 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:57:05.537058 20645 server_base.cc:1034] running on GCE node
I20260929 17:57:05.537647 20645 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:57:05.538856 20645 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:57:05.540019 20645 hybrid_clock.cc:648] HybridClock initialized: now 1790704625539999 us; error 34 us; skew 500 ppm
Sep 29 17:57:05 dist-test-slave-53nh krb5kdc[20632](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704625, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:57:05.542771 20645 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 17:57:05.543617 20645 webserver.cc:460] Webserver started at http://127.0.0.1:46853/ using document root <none> and password file <none>
I20260929 17:57:05.544132 20645 fs_manager.cc:359] Metadata directory not provided
I20260929 17:57:05.544188 20645 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:57:05.544351 20645 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:57:05.546444 20645 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790704375975685-392-0/minicluster-data/master-0/data/instance:
uuid: "9d2e7016d75b436d833269103e27dec1"
format_stamp: "Formatted at 2026-09-29 17:57:05 on dist-test-slave-53nh"
server_key: "3654d6b709c8760a8813e2d3c12e7716"
server_key_iv: "df32da7ee0d820dbe7ef6aa9d1323bfa"
server_key_version: "encryptionkey@0"
I20260929 17:57:05.546977 20645 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790704375975685-392-0/minicluster-data/master-0/wal/instance:
uuid: "9d2e7016d75b436d833269103e27dec1"
format_stamp: "Formatted at 2026-09-29 17:57:05 on dist-test-slave-53nh"
server_key: "3654d6b709c8760a8813e2d3c12e7716"
server_key_iv: "df32da7ee0d820dbe7ef6aa9d1323bfa"
server_key_version: "encryptionkey@0"
I20260929 17:57:05.550753 20645 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260929 17:57:05.553169 20660 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:57:05.554359 20645 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 17:57:05.554486 20645 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790704375975685-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790704375975685-392-0/minicluster-data/master-0/wal
uuid: "9d2e7016d75b436d833269103e27dec1"
format_stamp: "Formatted at 2026-09-29 17:57:05 on dist-test-slave-53nh"
server_key: "3654d6b709c8760a8813e2d3c12e7716"
server_key_iv: "df32da7ee0d820dbe7ef6aa9d1323bfa"
server_key_version: "encryptionkey@0"
I20260929 17:57:05.554596 20645 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790704375975685-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790704375975685-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790704375975685-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:57:05.574476 20645 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:57:05.577611 20645 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:57:05.577798 20645 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:57:05.584678 20645 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34143
I20260929 17:57:05.584703 20712 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34143 every 8 connection(s)
I20260929 17:57:05.585644 20645 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790704375975685-392-0/minicluster-data/master-0/data/info.pb
I20260929 17:57:05.588302 20713 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:57:05.590065 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskup3Soe/build/debug/bin/kudu as pid 20645
I20260929 17:57:05.590197 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790704375975685-392-0/minicluster-data/master-0/wal/instance
I20260929 17:57:05.590482 392 external_mini_cluster.cc:1428] Setting key 1c7efc9d23e25c20a239c8f9eb045d3c
I20260929 17:57:05.594619 20713 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d2e7016d75b436d833269103e27dec1: Bootstrap starting.
I20260929 17:57:05.596810 20713 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9d2e7016d75b436d833269103e27dec1: Neither blocks nor log segments found. Creating new log.
I20260929 17:57:05.597647 20713 log.cc:824] T 00000000000000000000000000000000 P 9d2e7016d75b436d833269103e27dec1: Log is configured to *not* fsync() on all Append() calls
Sep 29 17:57:05 dist-test-slave-53nh krb5kdc[20632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704625, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260929 17:57:05.599478 20713 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d2e7016d75b436d833269103e27dec1: No bootstrap required, opened a new log
I20260929 17:57:05.602890 20713 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9d2e7016d75b436d833269103e27dec1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d2e7016d75b436d833269103e27dec1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34143 } }
I20260929 17:57:05.603119 20713 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9d2e7016d75b436d833269103e27dec1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:57:05.603185 20713 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9d2e7016d75b436d833269103e27dec1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d2e7016d75b436d833269103e27dec1, State: Initialized, Role: FOLLOWER
I20260929 17:57:05.603613 20713 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9d2e7016d75b436d833269103e27dec1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d2e7016d75b436d833269103e27dec1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34143 } }
I20260929 17:57:05.603668 20716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:05.591914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45713 (local address 127.0.0.1:34143)
0929 17:57:05.592342 (+ 428us) server_negotiation.cc:206] Beginning negotiation
0929 17:57:05.592357 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 17:57:05.592392 (+ 35us) server_negotiation.cc:407] Connection header received
0929 17:57:05.593061 (+ 669us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:57:05.593080 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:57:05.593414 (+ 334us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:57:05.593707 (+ 293us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:57:05.594474 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:05.595467 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:05.596292 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:05.596762 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:05.596911 (+ 149us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:05.599292 (+ 2381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:57:05.599296 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:57:05.599309 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:57:05.599335 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:57:05.600862 (+ 1527us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:57:05.601370 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:57:05.601375 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:57:05.601379 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:57:05.601484 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:57:05.601775 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:57:05.601777 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:57:05.601778 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:57:05.602157 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:57:05.602355 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0929 17:57:05.602659 (+ 304us) server_negotiation.cc:299] Negotiation successful
0929 17:57:05.602857 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":292,"thread_start_us":130,"threads_started":1}
I20260929 17:57:05.603752 20713 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9d2e7016d75b436d833269103e27dec1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 17:57:05.603813 20713 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9d2e7016d75b436d833269103e27dec1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 17:57:05.603904 20713 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9d2e7016d75b436d833269103e27dec1 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:57:05.604741 20713 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9d2e7016d75b436d833269103e27dec1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d2e7016d75b436d833269103e27dec1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34143 } }
I20260929 17:57:05.605090 20713 leader_election.cc:302] T 00000000000000000000000000000000 P 9d2e7016d75b436d833269103e27dec1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9d2e7016d75b436d833269103e27dec1; no voters:
I20260929 17:57:05.605408 20713 leader_election.cc:288] T 00000000000000000000000000000000 P 9d2e7016d75b436d833269103e27dec1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 17:57:05.605742 20719 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9d2e7016d75b436d833269103e27dec1 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 17:57:05.605937 20719 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9d2e7016d75b436d833269103e27dec1 [term 1 LEADER]: Becoming Leader. State: Replica: 9d2e7016d75b436d833269103e27dec1, State: Running, Role: LEADER
I20260929 17:57:05.606228 20719 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9d2e7016d75b436d833269103e27dec1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d2e7016d75b436d833269103e27dec1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34143 } }
I20260929 17:57:05.606590 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 17:57:05.606678 20713 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9d2e7016d75b436d833269103e27dec1 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 17:57:05.607797 20718 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d2e7016d75b436d833269103e27dec1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9d2e7016d75b436d833269103e27dec1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d2e7016d75b436d833269103e27dec1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34143 } } }
I20260929 17:57:05.607939 20718 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d2e7016d75b436d833269103e27dec1 [sys.catalog]: This master's current role is: LEADER
I20260929 17:57:05.608237 20720 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d2e7016d75b436d833269103e27dec1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9d2e7016d75b436d833269103e27dec1. Latest consensus state: current_term: 1 leader_uuid: "9d2e7016d75b436d833269103e27dec1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d2e7016d75b436d833269103e27dec1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34143 } } }
I20260929 17:57:05.608326 20720 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d2e7016d75b436d833269103e27dec1 [sys.catalog]: This master's current role is: LEADER
W20260929 17:57:05.611366 20737 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9d2e7016d75b436d833269103e27dec1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 17:57:05.611469 20737 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 17:57:05.611526 20732 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 17:57:05.612249 20732 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 17:57:05.618746 20732 catalog_manager.cc:1347] Generated new cluster ID: 7bd457c2ec2d42908fd816fbcb367feb
I20260929 17:57:05.618831 20732 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 17:57:05.619232 20716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:05.609441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54603 (local address 127.0.0.1:34143)
0929 17:57:05.609538 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0929 17:57:05.609543 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:57:05.609948 (+ 405us) server_negotiation.cc:407] Connection header received
0929 17:57:05.610057 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:57:05.610060 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:57:05.610106 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:57:05.610188 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:57:05.610995 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:05.611723 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:05.612855 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:05.613220 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:05.613320 (+ 100us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:05.615024 (+ 1704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:57:05.615028 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:57:05.615031 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:57:05.615058 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:57:05.617148 (+ 2090us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:57:05.617698 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:57:05.617701 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:57:05.617703 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:57:05.617794 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:57:05.618141 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:57:05.618144 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:57:05.618146 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:57:05.618360 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:57:05.618509 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0929 17:57:05.619035 (+ 526us) server_negotiation.cc:299] Negotiation successful
0929 17:57:05.619119 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260929 17:57:05.627219 20732 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 17:57:05.628482 20732 catalog_manager.cc:1504] Loading token signing keys...
I20260929 17:57:05.641863 20732 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9d2e7016d75b436d833269103e27dec1: Generated new TSK 0
I20260929 17:57:05.642661 20732 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 17:57:05.653867 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskup3Soe/build/debug/bin/kudu with pid 20645
W20260929 17:57:05.658806 20630 connection.cc:535] client connection to 127.0.0.1:34143 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34143): Connection reset by peer (error 104)
2026-09-29T17:57:05Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1336 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790704375975685-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 17:57:05 dist-test-slave-53nh krb5kdc[20744](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 17:57:05 dist-test-slave-53nh krb5kdc[20744](info): set up 2 sockets
Sep 29 17:57:05 dist-test-slave-53nh krb5kdc[20744](info): commencing operation
krb5kdc: starting...
W20260929 17:57:06.604924 392 mini_kdc.cc:121] Time spent starting KDC: real 0.932s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790704375975685-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790704375975685-392-0/krb5kdc/test-user.keytab.
Sep 29 17:57:06 dist-test-slave-53nh krb5kdc[20744](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704626, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T17:57:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T17:57:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790704375975685-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790704375975685-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 17:57:06.725932 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskup3Soe/build/debug/bin/kudu
/tmp/dist-test-taskup3Soe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790704375975685-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790704375975685-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790704375975685-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790704375975685-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46057
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790704375975685-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46057
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790704375975685-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790704375975685-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790704375975685-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:57:06.837234 20757 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:57:06.837538 20757 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:57:06.837607 20757 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:57:06.841701 20757 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 17:57:06.841775 20757 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:57:06.841806 20757 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:57:06.841850 20757 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 17:57:06.841879 20757 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 17:57:06.847461 20757 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790704375975685-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790704375975685-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46057
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790704375975685-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46057
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790704375975685-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20757
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790704375975685-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:57:06.848596 20757 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:57:06.849766 20757 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:57:06.859318 20762 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:57:06.859407 20757 server_base.cc:1034] running on GCE node
W20260929 17:57:06.859334 20763 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:57:06.859318 20765 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:57:06.860277 20757 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:57:06.861835 20757 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:57:06.862975 20757 hybrid_clock.cc:648] HybridClock initialized: now 1790704626862939 us; error 59 us; skew 500 ppm
Sep 29 17:57:06 dist-test-slave-53nh krb5kdc[20744](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704626, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:57:06.867197 20757 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 17:57:06.868404 20757 webserver.cc:460] Webserver started at http://127.0.0.1:32825/ using document root <none> and password file <none>
I20260929 17:57:06.868968 20757 fs_manager.cc:359] Metadata directory not provided
I20260929 17:57:06.869076 20757 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:57:06.869270 20757 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:57:06.871330 20757 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790704375975685-392-0/minicluster-data/master-0/data/instance:
uuid: "f8b8cf4fa2e24bc4a337c3b8c53e3ca6"
format_stamp: "Formatted at 2026-09-29 17:57:06 on dist-test-slave-53nh"
server_key: "d6dc12196593793f9c535a0fa3e7b495"
server_key_iv: "ec7c67a385c2c6dacb90b850051880cb"
server_key_version: "encryptionkey@0"
I20260929 17:57:06.871872 20757 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790704375975685-392-0/minicluster-data/master-0/wal/instance:
uuid: "f8b8cf4fa2e24bc4a337c3b8c53e3ca6"
format_stamp: "Formatted at 2026-09-29 17:57:06 on dist-test-slave-53nh"
server_key: "d6dc12196593793f9c535a0fa3e7b495"
server_key_iv: "ec7c67a385c2c6dacb90b850051880cb"
server_key_version: "encryptionkey@0"
I20260929 17:57:06.875581 20757 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260929 17:57:06.877961 20772 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:57:06.879160 20757 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 17:57:06.879264 20757 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790704375975685-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790704375975685-392-0/minicluster-data/master-0/wal
uuid: "f8b8cf4fa2e24bc4a337c3b8c53e3ca6"
format_stamp: "Formatted at 2026-09-29 17:57:06 on dist-test-slave-53nh"
server_key: "d6dc12196593793f9c535a0fa3e7b495"
server_key_iv: "ec7c67a385c2c6dacb90b850051880cb"
server_key_version: "encryptionkey@0"
I20260929 17:57:06.879350 20757 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790704375975685-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790704375975685-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790704375975685-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:57:06.888833 20757 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:57:06.891820 20757 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:57:06.892000 20757 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:57:06.899075 20757 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46057
I20260929 17:57:06.899119 20824 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46057 every 8 connection(s)
I20260929 17:57:06.900095 20757 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790704375975685-392-0/minicluster-data/master-0/data/info.pb
I20260929 17:57:06.902125 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskup3Soe/build/debug/bin/kudu as pid 20757
I20260929 17:57:06.902261 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790704375975685-392-0/minicluster-data/master-0/wal/instance
I20260929 17:57:06.902539 392 external_mini_cluster.cc:1428] Setting key fcf638334fb95315b679702589cd9ebf
I20260929 17:57:06.903102 20825 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:57:06.909814 20825 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f8b8cf4fa2e24bc4a337c3b8c53e3ca6: Bootstrap starting.
I20260929 17:57:06.911839 20828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:06.904068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35615 (local address 127.0.0.1:46057)
0929 17:57:06.904532 (+ 464us) server_negotiation.cc:206] Beginning negotiation
0929 17:57:06.904547 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 17:57:06.904583 (+ 36us) server_negotiation.cc:407] Connection header received
0929 17:57:06.905295 (+ 712us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:57:06.905314 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:57:06.905661 (+ 347us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:57:06.905930 (+ 269us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:57:06.906308 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:06.907323 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:06.908504 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:06.908983 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:06.909154 (+ 171us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:06.910077 (+ 923us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:57:06.910082 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:57:06.910096 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 17:57:06.910498 (+ 402us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:57:06.910555 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 17:57:06.910578 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:57:06.910682 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0929 17:57:06.910914 (+ 232us) server_negotiation.cc:299] Negotiation successful
0929 17:57:06.911065 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300,"thread_start_us":142,"threads_started":1}
I20260929 17:57:06.912137 20825 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f8b8cf4fa2e24bc4a337c3b8c53e3ca6: Neither blocks nor log segments found. Creating new log.
I20260929 17:57:06.912814 20825 log.cc:824] T 00000000000000000000000000000000 P f8b8cf4fa2e24bc4a337c3b8c53e3ca6: Log is configured to *not* fsync() on all Append() calls
I20260929 17:57:06.914624 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 17:57:06.914783 20825 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f8b8cf4fa2e24bc4a337c3b8c53e3ca6: No bootstrap required, opened a new log
I20260929 17:57:06.918787 20825 raft_consensus.cc:348] T 00000000000000000000000000000000 P f8b8cf4fa2e24bc4a337c3b8c53e3ca6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8b8cf4fa2e24bc4a337c3b8c53e3ca6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46057 } }
I20260929 17:57:06.919067 20825 raft_consensus.cc:374] T 00000000000000000000000000000000 P f8b8cf4fa2e24bc4a337c3b8c53e3ca6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:57:06.919139 20825 raft_consensus.cc:729] T 00000000000000000000000000000000 P f8b8cf4fa2e24bc4a337c3b8c53e3ca6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8b8cf4fa2e24bc4a337c3b8c53e3ca6, State: Initialized, Role: FOLLOWER
I20260929 17:57:06.919538 20825 consensus_queue.cc:260] T 00000000000000000000000000000000 P f8b8cf4fa2e24bc4a337c3b8c53e3ca6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8b8cf4fa2e24bc4a337c3b8c53e3ca6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46057 } }
I20260929 17:57:06.919659 20825 raft_consensus.cc:388] T 00000000000000000000000000000000 P f8b8cf4fa2e24bc4a337c3b8c53e3ca6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 17:57:06.919721 20825 raft_consensus.cc:482] T 00000000000000000000000000000000 P f8b8cf4fa2e24bc4a337c3b8c53e3ca6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 17:57:06.919811 20825 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f8b8cf4fa2e24bc4a337c3b8c53e3ca6 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:57:06.920557 20828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:06.916504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52169 (local address 127.0.0.1:46057)
0929 17:57:06.916603 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0929 17:57:06.916606 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 17:57:06.916827 (+ 221us) server_negotiation.cc:407] Connection header received
0929 17:57:06.916982 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:57:06.916984 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:57:06.917053 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:57:06.917116 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:57:06.917372 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:06.917863 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:06.918911 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:06.919164 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:06.919250 (+ 86us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:06.920133 (+ 883us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:57:06.920137 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:57:06.920140 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 17:57:06.920152 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:57:06.920177 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 17:57:06.920187 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:57:06.920255 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0929 17:57:06.920391 (+ 136us) server_negotiation.cc:299] Negotiation successful
0929 17:57:06.920431 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260929 17:57:06.920744 20825 raft_consensus.cc:504] T 00000000000000000000000000000000 P f8b8cf4fa2e24bc4a337c3b8c53e3ca6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8b8cf4fa2e24bc4a337c3b8c53e3ca6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46057 } }
I20260929 17:57:06.921109 20825 leader_election.cc:302] T 00000000000000000000000000000000 P f8b8cf4fa2e24bc4a337c3b8c53e3ca6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f8b8cf4fa2e24bc4a337c3b8c53e3ca6; no voters:
I20260929 17:57:06.921397 20825 leader_election.cc:288] T 00000000000000000000000000000000 P f8b8cf4fa2e24bc4a337c3b8c53e3ca6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 17:57:06.921675 20835 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f8b8cf4fa2e24bc4a337c3b8c53e3ca6 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 17:57:06.921852 20835 raft_consensus.cc:686] T 00000000000000000000000000000000 P f8b8cf4fa2e24bc4a337c3b8c53e3ca6 [term 1 LEADER]: Becoming Leader. State: Replica: f8b8cf4fa2e24bc4a337c3b8c53e3ca6, State: Running, Role: LEADER
I20260929 17:57:06.922149 20835 consensus_queue.cc:237] T 00000000000000000000000000000000 P f8b8cf4fa2e24bc4a337c3b8c53e3ca6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8b8cf4fa2e24bc4a337c3b8c53e3ca6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46057 } }
I20260929 17:57:06.922700 20825 sys_catalog.cc:564] T 00000000000000000000000000000000 P f8b8cf4fa2e24bc4a337c3b8c53e3ca6 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 17:57:06.924009 20836 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8b8cf4fa2e24bc4a337c3b8c53e3ca6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f8b8cf4fa2e24bc4a337c3b8c53e3ca6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8b8cf4fa2e24bc4a337c3b8c53e3ca6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46057 } } }
I20260929 17:57:06.924044 20837 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8b8cf4fa2e24bc4a337c3b8c53e3ca6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f8b8cf4fa2e24bc4a337c3b8c53e3ca6. Latest consensus state: current_term: 1 leader_uuid: "f8b8cf4fa2e24bc4a337c3b8c53e3ca6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8b8cf4fa2e24bc4a337c3b8c53e3ca6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46057 } } }
I20260929 17:57:06.924129 20836 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8b8cf4fa2e24bc4a337c3b8c53e3ca6 [sys.catalog]: This master's current role is: LEADER
I20260929 17:57:06.924141 20837 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8b8cf4fa2e24bc4a337c3b8c53e3ca6 [sys.catalog]: This master's current role is: LEADER
I20260929 17:57:06.924579 20845 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 17:57:06.927490 20845 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 17:57:06.933147 20845 catalog_manager.cc:1347] Generated new cluster ID: 030ad04b93b241e6a7264a445fc3a87b
I20260929 17:57:06.933220 20845 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 17:57:06.940279 20845 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 17:57:06.941278 20845 catalog_manager.cc:1504] Loading token signing keys...
I20260929 17:57:06.953997 20845 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f8b8cf4fa2e24bc4a337c3b8c53e3ca6: Generated new TSK 0
I20260929 17:57:06.954664 20845 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 17:57:06.960270 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskup3Soe/build/debug/bin/kudu with pid 20757
W20260929 17:57:06.964928 20742 connection.cc:535] client connection to 127.0.0.1:46057 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46057): Connection reset by peer (error 104)
2026-09-29T17:57:06Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1306 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790704375975685-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 17:57:07 dist-test-slave-53nh krb5kdc[20856](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 17:57:07 dist-test-slave-53nh krb5kdc[20856](info): set up 2 sockets
Sep 29 17:57:07 dist-test-slave-53nh krb5kdc[20856](info): commencing operation
krb5kdc: starting...
W20260929 17:57:07.916918 392 mini_kdc.cc:121] Time spent starting KDC: real 0.938s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790704375975685-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790704375975685-392-0/krb5kdc/test-user.keytab.
Sep 29 17:57:07 dist-test-slave-53nh krb5kdc[20856](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704627, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T17:57:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T17:57:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790704375975685-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790704375975685-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 17:57:08.035579 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskup3Soe/build/debug/bin/kudu
/tmp/dist-test-taskup3Soe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790704375975685-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790704375975685-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790704375975685-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790704375975685-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37763
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790704375975685-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37763
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790704375975685-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790704375975685-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790704375975685-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:57:08.143746 20869 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:57:08.144021 20869 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:57:08.144064 20869 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:57:08.148530 20869 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 17:57:08.148612 20869 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:57:08.148631 20869 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:57:08.148646 20869 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 17:57:08.148660 20869 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 17:57:08.154052 20869 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790704375975685-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790704375975685-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37763
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790704375975685-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37763
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790704375975685-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20869
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790704375975685-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:57:08.155054 20869 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:57:08.155994 20869 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:57:08.164047 20874 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:57:08.164090 20875 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:57:08.164121 20877 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:57:08.164230 20869 server_base.cc:1034] running on GCE node
I20260929 17:57:08.164870 20869 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:57:08.166064 20869 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:57:08.167220 20869 hybrid_clock.cc:648] HybridClock initialized: now 1790704628167198 us; error 39 us; skew 500 ppm
Sep 29 17:57:08 dist-test-slave-53nh krb5kdc[20856](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704628, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:57:08.169970 20869 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 17:57:08.170998 20869 webserver.cc:460] Webserver started at http://127.0.0.1:33367/ using document root <none> and password file <none>
I20260929 17:57:08.171550 20869 fs_manager.cc:359] Metadata directory not provided
I20260929 17:57:08.171617 20869 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:57:08.171810 20869 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:57:08.174026 20869 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790704375975685-392-0/minicluster-data/master-0/data/instance:
uuid: "8880d45bca744e498cc6246149580558"
format_stamp: "Formatted at 2026-09-29 17:57:08 on dist-test-slave-53nh"
server_key: "872e1fcee0cf5a3071720307010bc209"
server_key_iv: "39f7f6579fe2be925a41a0d4921622d9"
server_key_version: "encryptionkey@0"
I20260929 17:57:08.174595 20869 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790704375975685-392-0/minicluster-data/master-0/wal/instance:
uuid: "8880d45bca744e498cc6246149580558"
format_stamp: "Formatted at 2026-09-29 17:57:08 on dist-test-slave-53nh"
server_key: "872e1fcee0cf5a3071720307010bc209"
server_key_iv: "39f7f6579fe2be925a41a0d4921622d9"
server_key_version: "encryptionkey@0"
I20260929 17:57:08.178515 20869 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 17:57:08.180943 20884 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:57:08.182255 20869 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 17:57:08.182365 20869 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790704375975685-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790704375975685-392-0/minicluster-data/master-0/wal
uuid: "8880d45bca744e498cc6246149580558"
format_stamp: "Formatted at 2026-09-29 17:57:08 on dist-test-slave-53nh"
server_key: "872e1fcee0cf5a3071720307010bc209"
server_key_iv: "39f7f6579fe2be925a41a0d4921622d9"
server_key_version: "encryptionkey@0"
I20260929 17:57:08.182464 20869 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790704375975685-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790704375975685-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790704375975685-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:57:08.186883 20869 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:57:08.189589 20869 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:57:08.189738 20869 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:57:08.196244 20869 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37763
I20260929 17:57:08.196269 20936 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37763 every 8 connection(s)
I20260929 17:57:08.197182 20869 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790704375975685-392-0/minicluster-data/master-0/data/info.pb
I20260929 17:57:08.199846 20937 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:57:08.201421 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskup3Soe/build/debug/bin/kudu as pid 20869
I20260929 17:57:08.201540 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790704375975685-392-0/minicluster-data/master-0/wal/instance
I20260929 17:57:08.201805 392 external_mini_cluster.cc:1428] Setting key ad0435e4cae5701a5b58292d2b21e823
I20260929 17:57:08.206475 20937 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8880d45bca744e498cc6246149580558: Bootstrap starting.
I20260929 17:57:08.207342 20940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:08.203141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43887 (local address 127.0.0.1:37763)
0929 17:57:08.203548 (+ 407us) server_negotiation.cc:206] Beginning negotiation
0929 17:57:08.203567 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0929 17:57:08.203610 (+ 43us) server_negotiation.cc:407] Connection header received
0929 17:57:08.204436 (+ 826us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:57:08.204461 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:57:08.204849 (+ 388us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:57:08.205271 (+ 422us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:57:08.205922 (+ 651us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:57:08.205927 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:57:08.205943 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 17:57:08.205959 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:57:08.206030 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 17:57:08.206056 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:57:08.206192 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0929 17:57:08.206258 (+ 66us) server_negotiation.cc:299] Negotiation successful
0929 17:57:08.206406 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":256,"thread_start_us":109,"threads_started":1}
I20260929 17:57:08.209146 20937 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8880d45bca744e498cc6246149580558: Neither blocks nor log segments found. Creating new log.
I20260929 17:57:08.209919 20937 log.cc:824] T 00000000000000000000000000000000 P 8880d45bca744e498cc6246149580558: Log is configured to *not* fsync() on all Append() calls
I20260929 17:57:08.210210 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 17:57:08.212277 20937 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8880d45bca744e498cc6246149580558: No bootstrap required, opened a new log
I20260929 17:57:08.214025 20940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:08.212432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33075 (local address 127.0.0.1:37763)
0929 17:57:08.212583 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0929 17:57:08.212587 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:57:08.212726 (+ 139us) server_negotiation.cc:407] Connection header received
0929 17:57:08.212853 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:57:08.212855 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:57:08.212886 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:57:08.212951 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:57:08.213676 (+ 725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:57:08.213679 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:57:08.213681 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 17:57:08.213692 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:57:08.213719 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 17:57:08.213729 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:57:08.213798 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0929 17:57:08.213878 (+ 80us) server_negotiation.cc:299] Negotiation successful
0929 17:57:08.213910 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260929 17:57:08.216356 20937 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8880d45bca744e498cc6246149580558 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8880d45bca744e498cc6246149580558" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37763 } }
I20260929 17:57:08.216615 20937 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8880d45bca744e498cc6246149580558 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:57:08.216670 20937 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8880d45bca744e498cc6246149580558 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8880d45bca744e498cc6246149580558, State: Initialized, Role: FOLLOWER
I20260929 17:57:08.217133 20937 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8880d45bca744e498cc6246149580558 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8880d45bca744e498cc6246149580558" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37763 } }
I20260929 17:57:08.217275 20937 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8880d45bca744e498cc6246149580558 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 17:57:08.217341 20937 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8880d45bca744e498cc6246149580558 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 17:57:08.217438 20937 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8880d45bca744e498cc6246149580558 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:57:08.218405 20937 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8880d45bca744e498cc6246149580558 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8880d45bca744e498cc6246149580558" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37763 } }
I20260929 17:57:08.218752 20937 leader_election.cc:302] T 00000000000000000000000000000000 P 8880d45bca744e498cc6246149580558 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8880d45bca744e498cc6246149580558; no voters:
I20260929 17:57:08.219067 20937 leader_election.cc:288] T 00000000000000000000000000000000 P 8880d45bca744e498cc6246149580558 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 17:57:08.219218 20947 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8880d45bca744e498cc6246149580558 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 17:57:08.219429 20947 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8880d45bca744e498cc6246149580558 [term 1 LEADER]: Becoming Leader. State: Replica: 8880d45bca744e498cc6246149580558, State: Running, Role: LEADER
I20260929 17:57:08.219741 20947 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8880d45bca744e498cc6246149580558 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8880d45bca744e498cc6246149580558" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37763 } }
I20260929 17:57:08.220280 20937 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8880d45bca744e498cc6246149580558 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 17:57:08.221290 20949 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8880d45bca744e498cc6246149580558 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8880d45bca744e498cc6246149580558. Latest consensus state: current_term: 1 leader_uuid: "8880d45bca744e498cc6246149580558" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8880d45bca744e498cc6246149580558" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37763 } } }
I20260929 17:57:08.221413 20949 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8880d45bca744e498cc6246149580558 [sys.catalog]: This master's current role is: LEADER
I20260929 17:57:08.221774 20956 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 17:57:08.221714 20948 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8880d45bca744e498cc6246149580558 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8880d45bca744e498cc6246149580558" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8880d45bca744e498cc6246149580558" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37763 } } }
I20260929 17:57:08.221828 20948 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8880d45bca744e498cc6246149580558 [sys.catalog]: This master's current role is: LEADER
I20260929 17:57:08.225118 20956 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 17:57:08.231815 20956 catalog_manager.cc:1347] Generated new cluster ID: 2b589c355fcd4082979e6f6d9c6be71e
I20260929 17:57:08.231901 20956 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 17:57:08.252076 20956 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 17:57:08.252430 20956 catalog_manager.cc:1504] Loading token signing keys...
I20260929 17:57:08.264719 20956 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8880d45bca744e498cc6246149580558: Generated new TSK 0
I20260929 17:57:08.265585 20956 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 17:57:08.269467 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskup3Soe/build/debug/bin/kudu with pid 20869
2026-09-29T17:57:08Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1309 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790704375975685-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 17:57:08 dist-test-slave-53nh krb5kdc[20968](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 17:57:08 dist-test-slave-53nh krb5kdc[20968](info): set up 2 sockets
Sep 29 17:57:08 dist-test-slave-53nh krb5kdc[20968](info): commencing operation
krb5kdc: starting...
W20260929 17:57:09.216962 392 mini_kdc.cc:121] Time spent starting KDC: real 0.929s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790704375975685-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790704375975685-392-0/krb5kdc/test-user.keytab.
Sep 29 17:57:09 dist-test-slave-53nh krb5kdc[20968](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704629, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T17:57:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T17:57:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790704375975685-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790704375975685-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 17:57:09.337550 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskup3Soe/build/debug/bin/kudu
/tmp/dist-test-taskup3Soe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790704375975685-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790704375975685-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790704375975685-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790704375975685-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34343
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790704375975685-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34343
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790704375975685-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790704375975685-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790704375975685-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:57:09.446563 20981 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:57:09.446848 20981 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:57:09.446908 20981 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:57:09.450927 20981 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 17:57:09.450997 20981 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:57:09.451023 20981 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:57:09.451046 20981 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 17:57:09.451076 20981 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 17:57:09.456557 20981 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790704375975685-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790704375975685-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34343
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790704375975685-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34343
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790704375975685-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20981
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790704375975685-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:57:09.457731 20981 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:57:09.458748 20981 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 17:57:09.467310 20981 server_base.cc:1034] running on GCE node
W20260929 17:57:09.467253 20989 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:57:09.467253 20986 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:57:09.467278 20987 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:57:09.468142 20981 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:57:09.469333 20981 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:57:09.470525 20981 hybrid_clock.cc:648] HybridClock initialized: now 1790704629470493 us; error 41 us; skew 500 ppm
Sep 29 17:57:09 dist-test-slave-53nh krb5kdc[20968](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704629, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:57:09.477697 20981 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 17:57:09.478745 20981 webserver.cc:460] Webserver started at http://127.0.0.1:41579/ using document root <none> and password file <none>
I20260929 17:57:09.479301 20981 fs_manager.cc:359] Metadata directory not provided
I20260929 17:57:09.479370 20981 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:57:09.479569 20981 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:57:09.481678 20981 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790704375975685-392-0/minicluster-data/master-0/data/instance:
uuid: "cc09033241de402a9575b9df9b4c5325"
format_stamp: "Formatted at 2026-09-29 17:57:09 on dist-test-slave-53nh"
server_key: "d427d04938dadcf2b628a44524cd656b"
server_key_iv: "b124a4d426f33a47b5f52accbb19b289"
server_key_version: "encryptionkey@0"
I20260929 17:57:09.482208 20981 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790704375975685-392-0/minicluster-data/master-0/wal/instance:
uuid: "cc09033241de402a9575b9df9b4c5325"
format_stamp: "Formatted at 2026-09-29 17:57:09 on dist-test-slave-53nh"
server_key: "d427d04938dadcf2b628a44524cd656b"
server_key_iv: "b124a4d426f33a47b5f52accbb19b289"
server_key_version: "encryptionkey@0"
I20260929 17:57:09.486074 20981 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 17:57:09.488736 20996 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:57:09.490129 20981 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 17:57:09.490257 20981 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790704375975685-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790704375975685-392-0/minicluster-data/master-0/wal
uuid: "cc09033241de402a9575b9df9b4c5325"
format_stamp: "Formatted at 2026-09-29 17:57:09 on dist-test-slave-53nh"
server_key: "d427d04938dadcf2b628a44524cd656b"
server_key_iv: "b124a4d426f33a47b5f52accbb19b289"
server_key_version: "encryptionkey@0"
I20260929 17:57:09.490370 20981 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790704375975685-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790704375975685-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790704375975685-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:57:09.501991 20981 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:57:09.505162 20981 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:57:09.505362 20981 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:57:09.512668 20981 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34343
I20260929 17:57:09.512737 21048 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34343 every 8 connection(s)
I20260929 17:57:09.513756 20981 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790704375975685-392-0/minicluster-data/master-0/data/info.pb
I20260929 17:57:09.516570 21049 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:57:09.522039 21049 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cc09033241de402a9575b9df9b4c5325: Bootstrap starting.
I20260929 17:57:09.523835 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskup3Soe/build/debug/bin/kudu as pid 20981
I20260929 17:57:09.523954 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790704375975685-392-0/minicluster-data/master-0/wal/instance
I20260929 17:57:09.524209 392 external_mini_cluster.cc:1428] Setting key fe0dfa6312f0f6d89c028e6f0ee74f41
I20260929 17:57:09.524509 21049 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cc09033241de402a9575b9df9b4c5325: Neither blocks nor log segments found. Creating new log.
I20260929 17:57:09.525399 21049 log.cc:824] T 00000000000000000000000000000000 P cc09033241de402a9575b9df9b4c5325: Log is configured to *not* fsync() on all Append() calls
I20260929 17:57:09.527326 21049 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cc09033241de402a9575b9df9b4c5325: No bootstrap required, opened a new log
I20260929 17:57:09.531572 21049 raft_consensus.cc:348] T 00000000000000000000000000000000 P cc09033241de402a9575b9df9b4c5325 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc09033241de402a9575b9df9b4c5325" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34343 } }
I20260929 17:57:09.531891 21049 raft_consensus.cc:374] T 00000000000000000000000000000000 P cc09033241de402a9575b9df9b4c5325 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:57:09.531955 21049 raft_consensus.cc:729] T 00000000000000000000000000000000 P cc09033241de402a9575b9df9b4c5325 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc09033241de402a9575b9df9b4c5325, State: Initialized, Role: FOLLOWER
I20260929 17:57:09.532505 21049 consensus_queue.cc:260] T 00000000000000000000000000000000 P cc09033241de402a9575b9df9b4c5325 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc09033241de402a9575b9df9b4c5325" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34343 } }
I20260929 17:57:09.532649 21049 raft_consensus.cc:388] T 00000000000000000000000000000000 P cc09033241de402a9575b9df9b4c5325 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 17:57:09.532711 21049 raft_consensus.cc:482] T 00000000000000000000000000000000 P cc09033241de402a9575b9df9b4c5325 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 17:57:09.532806 21049 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cc09033241de402a9575b9df9b4c5325 [term 0 FOLLOWER]: Advancing to term 1
Sep 29 17:57:09 dist-test-slave-53nh krb5kdc[20968](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704629, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260929 17:57:09.533815 21049 raft_consensus.cc:504] T 00000000000000000000000000000000 P cc09033241de402a9575b9df9b4c5325 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc09033241de402a9575b9df9b4c5325" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34343 } }
I20260929 17:57:09.534178 21049 leader_election.cc:302] T 00000000000000000000000000000000 P cc09033241de402a9575b9df9b4c5325 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cc09033241de402a9575b9df9b4c5325; no voters:
I20260929 17:57:09.534492 21049 leader_election.cc:288] T 00000000000000000000000000000000 P cc09033241de402a9575b9df9b4c5325 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 17:57:09.534631 21054 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cc09033241de402a9575b9df9b4c5325 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 17:57:09.534837 21054 raft_consensus.cc:686] T 00000000000000000000000000000000 P cc09033241de402a9575b9df9b4c5325 [term 1 LEADER]: Becoming Leader. State: Replica: cc09033241de402a9575b9df9b4c5325, State: Running, Role: LEADER
I20260929 17:57:09.535233 21054 consensus_queue.cc:237] T 00000000000000000000000000000000 P cc09033241de402a9575b9df9b4c5325 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc09033241de402a9575b9df9b4c5325" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34343 } }
I20260929 17:57:09.535827 21049 sys_catalog.cc:564] T 00000000000000000000000000000000 P cc09033241de402a9575b9df9b4c5325 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 17:57:09.536913 21055 sys_catalog.cc:455] T 00000000000000000000000000000000 P cc09033241de402a9575b9df9b4c5325 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cc09033241de402a9575b9df9b4c5325" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc09033241de402a9575b9df9b4c5325" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34343 } } }
I20260929 17:57:09.537087 21055 sys_catalog.cc:458] T 00000000000000000000000000000000 P cc09033241de402a9575b9df9b4c5325 [sys.catalog]: This master's current role is: LEADER
I20260929 17:57:09.537463 21056 sys_catalog.cc:455] T 00000000000000000000000000000000 P cc09033241de402a9575b9df9b4c5325 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cc09033241de402a9575b9df9b4c5325. Latest consensus state: current_term: 1 leader_uuid: "cc09033241de402a9575b9df9b4c5325" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc09033241de402a9575b9df9b4c5325" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34343 } } }
I20260929 17:57:09.537636 21056 sys_catalog.cc:458] T 00000000000000000000000000000000 P cc09033241de402a9575b9df9b4c5325 [sys.catalog]: This master's current role is: LEADER
I20260929 17:57:09.539548 21052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:09.525281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36983 (local address 127.0.0.1:34343)
0929 17:57:09.525685 (+ 404us) server_negotiation.cc:206] Beginning negotiation
0929 17:57:09.525704 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0929 17:57:09.525747 (+ 43us) server_negotiation.cc:407] Connection header received
0929 17:57:09.526487 (+ 740us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:57:09.526508 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:57:09.526871 (+ 363us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:57:09.527158 (+ 287us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:57:09.528017 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:09.529043 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:09.530199 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:09.530664 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:09.530778 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:09.533671 (+ 2893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:57:09.533676 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:57:09.533691 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:57:09.533730 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:57:09.535764 (+ 2034us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:57:09.536296 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:57:09.536302 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:57:09.536309 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:57:09.536455 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:57:09.536798 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:57:09.536802 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:57:09.536804 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:57:09.537435 (+ 631us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:57:09.537633 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0929 17:57:09.538520 (+ 887us) server_negotiation.cc:299] Negotiation successful
0929 17:57:09.538790 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":264,"thread_start_us":124,"threads_started":1}
W20260929 17:57:09.541158 21069 catalog_manager.cc:1558] T 00000000000000000000000000000000 P cc09033241de402a9575b9df9b4c5325: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 17:57:09.541232 21069 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 17:57:09.541293 21063 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 17:57:09.542111 21063 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 17:57:09.542829 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 17:57:09.553189 21052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:09.545434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32965 (local address 127.0.0.1:34343)
0929 17:57:09.545555 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0929 17:57:09.545560 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:57:09.545850 (+ 290us) server_negotiation.cc:407] Connection header received
0929 17:57:09.545968 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:57:09.545971 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:57:09.546026 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:57:09.546087 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:57:09.546904 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:09.547439 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:09.548241 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:09.548490 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:09.548578 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:09.550218 (+ 1640us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:57:09.550222 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:57:09.550225 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:57:09.550253 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:57:09.551555 (+ 1302us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:57:09.552004 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:57:09.552006 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:57:09.552008 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:57:09.552131 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:57:09.552370 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:57:09.552372 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:57:09.552373 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:57:09.552557 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:57:09.552660 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0929 17:57:09.552866 (+ 206us) server_negotiation.cc:299] Negotiation successful
0929 17:57:09.552962 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260929 17:57:09.554785 21063 catalog_manager.cc:1347] Generated new cluster ID: b0a74b2153e04431b6744d9f0de44e9e
I20260929 17:57:09.554858 21063 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 17:57:09.562525 21063 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 17:57:09.563486 21063 catalog_manager.cc:1504] Loading token signing keys...
I20260929 17:57:09.574142 21063 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cc09033241de402a9575b9df9b4c5325: Generated new TSK 0
I20260929 17:57:09.574934 21063 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 17:57:09.581391 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskup3Soe/build/debug/bin/kudu with pid 20981
2026-09-29T17:57:09Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1311 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790704375975685-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 17:57:09 dist-test-slave-53nh krb5kdc[21080](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 17:57:09 dist-test-slave-53nh krb5kdc[21080](info): set up 2 sockets
Sep 29 17:57:09 dist-test-slave-53nh krb5kdc[21080](info): commencing operation
krb5kdc: starting...
W20260929 17:57:10.537117 392 mini_kdc.cc:121] Time spent starting KDC: real 0.937s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790704375975685-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790704375975685-392-0/krb5kdc/test-user.keytab.
Sep 29 17:57:10 dist-test-slave-53nh krb5kdc[21080](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704630, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T17:57:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T17:57:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790704375975685-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790704375975685-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 17:57:10.655427 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskup3Soe/build/debug/bin/kudu
/tmp/dist-test-taskup3Soe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790704375975685-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790704375975685-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790704375975685-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790704375975685-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39449
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790704375975685-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39449
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790704375975685-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790704375975685-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790704375975685-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:57:10.764336 21093 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:57:10.764604 21093 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:57:10.764647 21093 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:57:10.768651 21093 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 17:57:10.768715 21093 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:57:10.768733 21093 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:57:10.768755 21093 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 17:57:10.768774 21093 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 17:57:10.774194 21093 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790704375975685-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790704375975685-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39449
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790704375975685-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39449
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790704375975685-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21093
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790704375975685-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:57:10.775312 21093 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:57:10.776356 21093 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:57:10.784713 21098 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:57:10.784699 21101 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:57:10.784696 21099 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:57:10.784839 21093 server_base.cc:1034] running on GCE node
I20260929 17:57:10.785528 21093 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:57:10.786612 21093 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:57:10.787786 21093 hybrid_clock.cc:648] HybridClock initialized: now 1790704630787760 us; error 37 us; skew 500 ppm
Sep 29 17:57:10 dist-test-slave-53nh krb5kdc[21080](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704630, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:57:10.790488 21093 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 17:57:10.791354 21093 webserver.cc:460] Webserver started at http://127.0.0.1:38545/ using document root <none> and password file <none>
I20260929 17:57:10.791888 21093 fs_manager.cc:359] Metadata directory not provided
I20260929 17:57:10.791952 21093 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:57:10.792191 21093 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:57:10.794261 21093 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790704375975685-392-0/minicluster-data/master-0/data/instance:
uuid: "3603055f93cc4c27a7ba5a30d2bec5da"
format_stamp: "Formatted at 2026-09-29 17:57:10 on dist-test-slave-53nh"
server_key: "9ba1b72978edcecef77e9ff9467b8b2a"
server_key_iv: "9fbe87508c16d059826bdc99fabb4734"
server_key_version: "encryptionkey@0"
I20260929 17:57:10.794785 21093 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790704375975685-392-0/minicluster-data/master-0/wal/instance:
uuid: "3603055f93cc4c27a7ba5a30d2bec5da"
format_stamp: "Formatted at 2026-09-29 17:57:10 on dist-test-slave-53nh"
server_key: "9ba1b72978edcecef77e9ff9467b8b2a"
server_key_iv: "9fbe87508c16d059826bdc99fabb4734"
server_key_version: "encryptionkey@0"
I20260929 17:57:10.798420 21093 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 17:57:10.800905 21108 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:57:10.802223 21093 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 17:57:10.802335 21093 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790704375975685-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790704375975685-392-0/minicluster-data/master-0/wal
uuid: "3603055f93cc4c27a7ba5a30d2bec5da"
format_stamp: "Formatted at 2026-09-29 17:57:10 on dist-test-slave-53nh"
server_key: "9ba1b72978edcecef77e9ff9467b8b2a"
server_key_iv: "9fbe87508c16d059826bdc99fabb4734"
server_key_version: "encryptionkey@0"
I20260929 17:57:10.802440 21093 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790704375975685-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790704375975685-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790704375975685-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:57:10.829768 21093 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:57:10.832787 21093 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:57:10.832965 21093 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:57:10.839643 21093 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39449
I20260929 17:57:10.839673 21160 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39449 every 8 connection(s)
I20260929 17:57:10.840579 21093 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790704375975685-392-0/minicluster-data/master-0/data/info.pb
I20260929 17:57:10.841733 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskup3Soe/build/debug/bin/kudu as pid 21093
I20260929 17:57:10.841886 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790704375975685-392-0/minicluster-data/master-0/wal/instance
I20260929 17:57:10.842175 392 external_mini_cluster.cc:1428] Setting key b18b9d0352c7e4e4dd54b5d36c51a100
I20260929 17:57:10.843854 21161 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:57:10.849444 21161 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3603055f93cc4c27a7ba5a30d2bec5da: Bootstrap starting.
I20260929 17:57:10.850697 21164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:10.843618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50737 (local address 127.0.0.1:39449)
0929 17:57:10.844066 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0929 17:57:10.844081 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 17:57:10.844115 (+ 34us) server_negotiation.cc:407] Connection header received
0929 17:57:10.844762 (+ 647us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:57:10.844779 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:57:10.845151 (+ 372us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:57:10.845463 (+ 312us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:57:10.845853 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:10.846834 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:10.847694 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:10.848141 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:10.848302 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:10.849109 (+ 807us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:57:10.849114 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:57:10.849126 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 17:57:10.849504 (+ 378us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:57:10.849569 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 17:57:10.849598 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:57:10.849679 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0929 17:57:10.849805 (+ 126us) server_negotiation.cc:299] Negotiation successful
0929 17:57:10.850009 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300,"thread_start_us":143,"threads_started":1}
I20260929 17:57:10.851719 21161 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3603055f93cc4c27a7ba5a30d2bec5da: Neither blocks nor log segments found. Creating new log.
I20260929 17:57:10.852456 21161 log.cc:824] T 00000000000000000000000000000000 P 3603055f93cc4c27a7ba5a30d2bec5da: Log is configured to *not* fsync() on all Append() calls
I20260929 17:57:10.853327 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 17:57:10.854763 21161 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3603055f93cc4c27a7ba5a30d2bec5da: No bootstrap required, opened a new log
I20260929 17:57:10.859258 21161 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3603055f93cc4c27a7ba5a30d2bec5da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3603055f93cc4c27a7ba5a30d2bec5da" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39449 } }
I20260929 17:57:10.859596 21161 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3603055f93cc4c27a7ba5a30d2bec5da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:57:10.859640 21164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:10.855630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35819 (local address 127.0.0.1:39449)
0929 17:57:10.855748 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0929 17:57:10.855752 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:57:10.855887 (+ 135us) server_negotiation.cc:407] Connection header received
0929 17:57:10.855998 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:57:10.856000 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:57:10.856047 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:57:10.856118 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:57:10.856412 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:10.856954 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:10.858095 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:10.858340 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:10.858419 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:10.859217 (+ 798us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:57:10.859220 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:57:10.859222 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 17:57:10.859235 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:57:10.859258 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 17:57:10.859268 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:57:10.859320 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0929 17:57:10.859424 (+ 104us) server_negotiation.cc:299] Negotiation successful
0929 17:57:10.859504 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 17:57:10.859659 21161 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3603055f93cc4c27a7ba5a30d2bec5da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3603055f93cc4c27a7ba5a30d2bec5da, State: Initialized, Role: FOLLOWER
I20260929 17:57:10.860126 21161 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3603055f93cc4c27a7ba5a30d2bec5da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3603055f93cc4c27a7ba5a30d2bec5da" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39449 } }
I20260929 17:57:10.860262 21161 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3603055f93cc4c27a7ba5a30d2bec5da [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 17:57:10.860327 21161 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3603055f93cc4c27a7ba5a30d2bec5da [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 17:57:10.860419 21161 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3603055f93cc4c27a7ba5a30d2bec5da [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:57:10.861410 21161 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3603055f93cc4c27a7ba5a30d2bec5da [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3603055f93cc4c27a7ba5a30d2bec5da" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39449 } }
I20260929 17:57:10.861856 21161 leader_election.cc:302] T 00000000000000000000000000000000 P 3603055f93cc4c27a7ba5a30d2bec5da [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3603055f93cc4c27a7ba5a30d2bec5da; no voters:
I20260929 17:57:10.862210 21161 leader_election.cc:288] T 00000000000000000000000000000000 P 3603055f93cc4c27a7ba5a30d2bec5da [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 17:57:10.862332 21171 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3603055f93cc4c27a7ba5a30d2bec5da [term 1 FOLLOWER]: Leader election won for term 1
I20260929 17:57:10.862566 21171 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3603055f93cc4c27a7ba5a30d2bec5da [term 1 LEADER]: Becoming Leader. State: Replica: 3603055f93cc4c27a7ba5a30d2bec5da, State: Running, Role: LEADER
I20260929 17:57:10.862890 21171 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3603055f93cc4c27a7ba5a30d2bec5da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3603055f93cc4c27a7ba5a30d2bec5da" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39449 } }
I20260929 17:57:10.863456 21161 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3603055f93cc4c27a7ba5a30d2bec5da [sys.catalog]: configured and running, proceeding with master startup.
I20260929 17:57:10.864604 21172 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3603055f93cc4c27a7ba5a30d2bec5da [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3603055f93cc4c27a7ba5a30d2bec5da" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3603055f93cc4c27a7ba5a30d2bec5da" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39449 } } }
I20260929 17:57:10.864614 21173 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3603055f93cc4c27a7ba5a30d2bec5da [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3603055f93cc4c27a7ba5a30d2bec5da. Latest consensus state: current_term: 1 leader_uuid: "3603055f93cc4c27a7ba5a30d2bec5da" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3603055f93cc4c27a7ba5a30d2bec5da" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39449 } } }
I20260929 17:57:10.864753 21172 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3603055f93cc4c27a7ba5a30d2bec5da [sys.catalog]: This master's current role is: LEADER
I20260929 17:57:10.864754 21173 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3603055f93cc4c27a7ba5a30d2bec5da [sys.catalog]: This master's current role is: LEADER
I20260929 17:57:10.865175 21180 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 17:57:10.867873 21180 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 17:57:10.873152 21180 catalog_manager.cc:1347] Generated new cluster ID: c4130041b762427ab288e34eb775f48d
I20260929 17:57:10.873222 21180 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 17:57:10.896503 21180 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 17:57:10.897995 21180 catalog_manager.cc:1504] Loading token signing keys...
I20260929 17:57:10.908067 21180 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3603055f93cc4c27a7ba5a30d2bec5da: Generated new TSK 0
I20260929 17:57:10.908990 21180 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 17:57:10.915335 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskup3Soe/build/debug/bin/kudu with pid 21093
2026-09-29T17:57:10Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1333 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790704375975685-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 17:57:10 dist-test-slave-53nh krb5kdc[21192](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 17:57:10 dist-test-slave-53nh krb5kdc[21192](info): set up 2 sockets
Sep 29 17:57:10 dist-test-slave-53nh krb5kdc[21192](info): commencing operation
krb5kdc: starting...
W20260929 17:57:11.873584 392 mini_kdc.cc:121] Time spent starting KDC: real 0.939s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790704375975685-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790704375975685-392-0/krb5kdc/test-user.keytab.
Sep 29 17:57:11 dist-test-slave-53nh krb5kdc[21192](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704631, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T17:57:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T17:57:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790704375975685-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790704375975685-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 17:57:11.997741 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskup3Soe/build/debug/bin/kudu
/tmp/dist-test-taskup3Soe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790704375975685-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790704375975685-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790704375975685-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790704375975685-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42425
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790704375975685-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36507
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42425
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790704375975685-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790704375975685-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790704375975685-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:57:12.110168 21205 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:57:12.110463 21205 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:57:12.110507 21205 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:57:12.114774 21205 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 17:57:12.114856 21205 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:57:12.114876 21205 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:57:12.114893 21205 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 17:57:12.114908 21205 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 17:57:12.120545 21205 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36507
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790704375975685-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790704375975685-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42425
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790704375975685-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42425
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790704375975685-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21205
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790704375975685-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:57:12.121699 21205 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:57:12.122743 21205 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:57:12.131297 21211 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:57:12.131326 21210 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:57:12.131558 21205 server_base.cc:1034] running on GCE node
W20260929 17:57:12.131300 21213 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:57:12.132323 21205 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:57:12.133646 21205 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:57:12.134809 21205 hybrid_clock.cc:648] HybridClock initialized: now 1790704632134782 us; error 48 us; skew 500 ppm
Sep 29 17:57:12 dist-test-slave-53nh krb5kdc[21192](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704632, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:57:12.137858 21205 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 17:57:12.138906 21205 webserver.cc:460] Webserver started at http://127.0.0.1:41519/ using document root <none> and password file <none>
I20260929 17:57:12.139535 21205 fs_manager.cc:359] Metadata directory not provided
I20260929 17:57:12.139601 21205 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:57:12.139808 21205 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:57:12.142015 21205 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790704375975685-392-0/minicluster-data/master-0/data/instance:
uuid: "8078ee62df7c440888f839f15f84f445"
format_stamp: "Formatted at 2026-09-29 17:57:12 on dist-test-slave-53nh"
server_key: "5a8e6300d5a669bf3183a023b2120391"
server_key_iv: "6c5fc600201fc8e77cd8bd3af0e3103b"
server_key_version: "encryptionkey@0"
I20260929 17:57:12.142580 21205 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790704375975685-392-0/minicluster-data/master-0/wal/instance:
uuid: "8078ee62df7c440888f839f15f84f445"
format_stamp: "Formatted at 2026-09-29 17:57:12 on dist-test-slave-53nh"
server_key: "5a8e6300d5a669bf3183a023b2120391"
server_key_iv: "6c5fc600201fc8e77cd8bd3af0e3103b"
server_key_version: "encryptionkey@0"
I20260929 17:57:12.146600 21205 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 17:57:12.149292 21220 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:57:12.150669 21205 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 17:57:12.150803 21205 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790704375975685-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790704375975685-392-0/minicluster-data/master-0/wal
uuid: "8078ee62df7c440888f839f15f84f445"
format_stamp: "Formatted at 2026-09-29 17:57:12 on dist-test-slave-53nh"
server_key: "5a8e6300d5a669bf3183a023b2120391"
server_key_iv: "6c5fc600201fc8e77cd8bd3af0e3103b"
server_key_version: "encryptionkey@0"
I20260929 17:57:12.150921 21205 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790704375975685-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790704375975685-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790704375975685-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:57:12.155665 21205 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:57:12.158779 21205 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:57:12.158951 21205 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:57:12.165584 21205 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42425
I20260929 17:57:12.165596 21272 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42425 every 8 connection(s)
I20260929 17:57:12.166498 21205 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790704375975685-392-0/minicluster-data/master-0/data/info.pb
I20260929 17:57:12.169240 21273 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:57:12.174196 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskup3Soe/build/debug/bin/kudu as pid 21205
I20260929 17:57:12.174316 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790704375975685-392-0/minicluster-data/master-0/wal/instance
I20260929 17:57:12.174561 392 external_mini_cluster.cc:1428] Setting key 70a4492aff8c43951ba98a09983829bb
I20260929 17:57:12.174957 21273 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8078ee62df7c440888f839f15f84f445: Bootstrap starting.
I20260929 17:57:12.177569 21273 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8078ee62df7c440888f839f15f84f445: Neither blocks nor log segments found. Creating new log.
I20260929 17:57:12.178453 21273 log.cc:824] T 00000000000000000000000000000000 P 8078ee62df7c440888f839f15f84f445: Log is configured to *not* fsync() on all Append() calls
I20260929 17:57:12.180028 21276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:12.175996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40181 (local address 127.0.0.1:42425)
0929 17:57:12.176452 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0929 17:57:12.176469 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 17:57:12.176507 (+ 38us) server_negotiation.cc:407] Connection header received
0929 17:57:12.177254 (+ 747us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:57:12.177275 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:57:12.177602 (+ 327us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:57:12.177939 (+ 337us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:57:12.178792 (+ 853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:57:12.178797 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:57:12.178809 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 17:57:12.178820 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:57:12.178874 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 17:57:12.178893 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:57:12.178974 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0929 17:57:12.179112 (+ 138us) server_negotiation.cc:299] Negotiation successful
0929 17:57:12.179261 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":292,"thread_start_us":129,"threads_started":1}
I20260929 17:57:12.180485 21273 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8078ee62df7c440888f839f15f84f445: No bootstrap required, opened a new log
I20260929 17:57:12.182842 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 17:57:12.184561 21273 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8078ee62df7c440888f839f15f84f445 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8078ee62df7c440888f839f15f84f445" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42425 } }
I20260929 17:57:12.184881 21273 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8078ee62df7c440888f839f15f84f445 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:57:12.184940 21273 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8078ee62df7c440888f839f15f84f445 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8078ee62df7c440888f839f15f84f445, State: Initialized, Role: FOLLOWER
I20260929 17:57:12.185428 21273 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8078ee62df7c440888f839f15f84f445 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8078ee62df7c440888f839f15f84f445" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42425 } }
I20260929 17:57:12.185563 21273 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8078ee62df7c440888f839f15f84f445 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 17:57:12.185621 21273 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8078ee62df7c440888f839f15f84f445 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 17:57:12.185708 21273 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8078ee62df7c440888f839f15f84f445 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:57:12.186633 21273 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8078ee62df7c440888f839f15f84f445 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8078ee62df7c440888f839f15f84f445" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42425 } }
I20260929 17:57:12.187041 21276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:12.185254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38739 (local address 127.0.0.1:42425)
0929 17:57:12.185383 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0929 17:57:12.185387 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:57:12.185626 (+ 239us) server_negotiation.cc:407] Connection header received
0929 17:57:12.185766 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:57:12.185769 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:57:12.185811 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:57:12.185890 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:57:12.186632 (+ 742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:57:12.186636 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:57:12.186639 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 17:57:12.186652 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:57:12.186684 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 17:57:12.186697 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:57:12.186776 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0929 17:57:12.186858 (+ 82us) server_negotiation.cc:299] Negotiation successful
0929 17:57:12.186910 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 17:57:12.187002 21273 leader_election.cc:302] T 00000000000000000000000000000000 P 8078ee62df7c440888f839f15f84f445 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8078ee62df7c440888f839f15f84f445; no voters:
I20260929 17:57:12.187325 21273 leader_election.cc:288] T 00000000000000000000000000000000 P 8078ee62df7c440888f839f15f84f445 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 17:57:12.187427 21283 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8078ee62df7c440888f839f15f84f445 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 17:57:12.187649 21283 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8078ee62df7c440888f839f15f84f445 [term 1 LEADER]: Becoming Leader. State: Replica: 8078ee62df7c440888f839f15f84f445, State: Running, Role: LEADER
I20260929 17:57:12.188032 21283 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8078ee62df7c440888f839f15f84f445 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8078ee62df7c440888f839f15f84f445" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42425 } }
I20260929 17:57:12.188541 21273 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8078ee62df7c440888f839f15f84f445 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 17:57:12.189689 21285 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8078ee62df7c440888f839f15f84f445 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8078ee62df7c440888f839f15f84f445" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8078ee62df7c440888f839f15f84f445" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42425 } } }
I20260929 17:57:12.189805 21285 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8078ee62df7c440888f839f15f84f445 [sys.catalog]: This master's current role is: LEADER
I20260929 17:57:12.190146 21284 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8078ee62df7c440888f839f15f84f445 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8078ee62df7c440888f839f15f84f445. Latest consensus state: current_term: 1 leader_uuid: "8078ee62df7c440888f839f15f84f445" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8078ee62df7c440888f839f15f84f445" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42425 } } }
I20260929 17:57:12.190205 21292 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 17:57:12.190313 21284 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8078ee62df7c440888f839f15f84f445 [sys.catalog]: This master's current role is: LEADER
I20260929 17:57:12.193521 21292 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 17:57:12.199123 21292 catalog_manager.cc:1347] Generated new cluster ID: 755c0ab6adda4307b0746eef35f08b03
I20260929 17:57:12.199234 21292 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 17:57:12.214308 21292 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 17:57:12.214746 21292 catalog_manager.cc:1504] Loading token signing keys...
I20260929 17:57:12.235339 21292 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8078ee62df7c440888f839f15f84f445: Generated new TSK 0
I20260929 17:57:12.236219 21292 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 17:57:12.249323 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskup3Soe/build/debug/bin/kudu with pid 21205
2026-09-29T17:57:12Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1334 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790704375975685-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 17:57:12 dist-test-slave-53nh krb5kdc[21304](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 17:57:12 dist-test-slave-53nh krb5kdc[21304](info): set up 2 sockets
Sep 29 17:57:12 dist-test-slave-53nh krb5kdc[21304](info): commencing operation
krb5kdc: starting...
W20260929 17:57:13.200932 392 mini_kdc.cc:121] Time spent starting KDC: real 0.932s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790704375975685-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790704375975685-392-0/krb5kdc/test-user.keytab.
Sep 29 17:57:13 dist-test-slave-53nh krb5kdc[21304](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704633, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T17:57:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T17:57:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790704375975685-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790704375975685-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 17:57:13.320092 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskup3Soe/build/debug/bin/kudu
/tmp/dist-test-taskup3Soe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790704375975685-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790704375975685-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790704375975685-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790704375975685-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43073
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790704375975685-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43073
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790704375975685-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790704375975685-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790704375975685-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:57:13.429217 21317 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:57:13.429534 21317 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:57:13.429597 21317 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:57:13.433791 21317 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 17:57:13.433876 21317 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:57:13.433904 21317 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:57:13.433928 21317 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 17:57:13.433946 21317 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 17:57:13.439540 21317 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790704375975685-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790704375975685-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43073
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790704375975685-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43073
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790704375975685-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21317
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790704375975685-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:57:13.440713 21317 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:57:13.441838 21317 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:57:13.450469 21325 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:57:13.450472 21323 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:57:13.450470 21322 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:57:13.450839 21317 server_base.cc:1034] running on GCE node
I20260929 17:57:13.451444 21317 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:57:13.452623 21317 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:57:13.453783 21317 hybrid_clock.cc:648] HybridClock initialized: now 1790704633453778 us; error 43 us; skew 500 ppm
Sep 29 17:57:13 dist-test-slave-53nh krb5kdc[21304](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704633, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:57:13.456881 21317 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 17:57:13.457943 21317 webserver.cc:460] Webserver started at http://127.0.0.1:39427/ using document root <none> and password file <none>
I20260929 17:57:13.458570 21317 fs_manager.cc:359] Metadata directory not provided
I20260929 17:57:13.458648 21317 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:57:13.458866 21317 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:57:13.460958 21317 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790704375975685-392-0/minicluster-data/master-0/data/instance:
uuid: "793649a9666f4e89aa24efe161541746"
format_stamp: "Formatted at 2026-09-29 17:57:13 on dist-test-slave-53nh"
server_key: "5297b472cf00979cc6f228c43da3e895"
server_key_iv: "648e12ccf7f5e61272fa75b2bf30c2ca"
server_key_version: "encryptionkey@0"
I20260929 17:57:13.461593 21317 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790704375975685-392-0/minicluster-data/master-0/wal/instance:
uuid: "793649a9666f4e89aa24efe161541746"
format_stamp: "Formatted at 2026-09-29 17:57:13 on dist-test-slave-53nh"
server_key: "5297b472cf00979cc6f228c43da3e895"
server_key_iv: "648e12ccf7f5e61272fa75b2bf30c2ca"
server_key_version: "encryptionkey@0"
I20260929 17:57:13.465512 21317 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260929 17:57:13.468062 21332 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:57:13.469520 21317 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 17:57:13.469643 21317 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790704375975685-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790704375975685-392-0/minicluster-data/master-0/wal
uuid: "793649a9666f4e89aa24efe161541746"
format_stamp: "Formatted at 2026-09-29 17:57:13 on dist-test-slave-53nh"
server_key: "5297b472cf00979cc6f228c43da3e895"
server_key_iv: "648e12ccf7f5e61272fa75b2bf30c2ca"
server_key_version: "encryptionkey@0"
I20260929 17:57:13.469753 21317 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790704375975685-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790704375975685-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790704375975685-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:57:13.501834 21317 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:57:13.505211 21317 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:57:13.505400 21317 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:57:13.512627 21317 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43073
I20260929 17:57:13.512656 21384 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43073 every 8 connection(s)
I20260929 17:57:13.513656 21317 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790704375975685-392-0/minicluster-data/master-0/data/info.pb
I20260929 17:57:13.516583 21385 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:57:13.516772 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskup3Soe/build/debug/bin/kudu as pid 21317
I20260929 17:57:13.516889 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790704375975685-392-0/minicluster-data/master-0/wal/instance
I20260929 17:57:13.517184 392 external_mini_cluster.cc:1428] Setting key 78bd9e58e52abdb6ecd802ee1789c2bf
I20260929 17:57:13.523504 21385 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 793649a9666f4e89aa24efe161541746: Bootstrap starting.
I20260929 17:57:13.526108 21385 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 793649a9666f4e89aa24efe161541746: Neither blocks nor log segments found. Creating new log.
Sep 29 17:57:13 dist-test-slave-53nh krb5kdc[21304](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704633, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260929 17:57:13.526805 21385 log.cc:824] T 00000000000000000000000000000000 P 793649a9666f4e89aa24efe161541746: Log is configured to *not* fsync() on all Append() calls
I20260929 17:57:13.529179 21385 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 793649a9666f4e89aa24efe161541746: No bootstrap required, opened a new log
I20260929 17:57:13.531992 21388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:13.518672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.108:43657 (local address 127.0.0.1:43073)
0929 17:57:13.519068 (+ 396us) server_negotiation.cc:206] Beginning negotiation
0929 17:57:13.519084 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 17:57:13.519119 (+ 35us) server_negotiation.cc:407] Connection header received
0929 17:57:13.519780 (+ 661us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:57:13.519797 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:57:13.520158 (+ 361us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:57:13.520446 (+ 288us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:57:13.521335 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:13.522337 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:13.523471 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:13.523964 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:13.524078 (+ 114us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:13.526755 (+ 2677us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:57:13.526760 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:57:13.526774 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:57:13.526805 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:57:13.528729 (+ 1924us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:57:13.529338 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:57:13.529344 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:57:13.529350 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:57:13.529492 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:57:13.529805 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:57:13.529808 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:57:13.529810 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:57:13.530307 (+ 497us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:57:13.530511 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0929 17:57:13.530768 (+ 257us) server_negotiation.cc:299] Negotiation successful
0929 17:57:13.530997 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":261,"thread_start_us":110,"threads_started":1}
I20260929 17:57:13.533542 21385 raft_consensus.cc:348] T 00000000000000000000000000000000 P 793649a9666f4e89aa24efe161541746 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "793649a9666f4e89aa24efe161541746" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43073 } }
I20260929 17:57:13.533828 21385 raft_consensus.cc:374] T 00000000000000000000000000000000 P 793649a9666f4e89aa24efe161541746 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:57:13.533880 21385 raft_consensus.cc:729] T 00000000000000000000000000000000 P 793649a9666f4e89aa24efe161541746 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 793649a9666f4e89aa24efe161541746, State: Initialized, Role: FOLLOWER
I20260929 17:57:13.534329 21385 consensus_queue.cc:260] T 00000000000000000000000000000000 P 793649a9666f4e89aa24efe161541746 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "793649a9666f4e89aa24efe161541746" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43073 } }
I20260929 17:57:13.534461 21385 raft_consensus.cc:388] T 00000000000000000000000000000000 P 793649a9666f4e89aa24efe161541746 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 17:57:13.534523 21385 raft_consensus.cc:482] T 00000000000000000000000000000000 P 793649a9666f4e89aa24efe161541746 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 17:57:13.534606 21385 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 793649a9666f4e89aa24efe161541746 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:57:13.534986 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 17:57:13.535724 21385 raft_consensus.cc:504] T 00000000000000000000000000000000 P 793649a9666f4e89aa24efe161541746 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "793649a9666f4e89aa24efe161541746" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43073 } }
I20260929 17:57:13.536110 21385 leader_election.cc:302] T 00000000000000000000000000000000 P 793649a9666f4e89aa24efe161541746 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 793649a9666f4e89aa24efe161541746; no voters:
I20260929 17:57:13.536470 21385 leader_election.cc:288] T 00000000000000000000000000000000 P 793649a9666f4e89aa24efe161541746 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 17:57:13.536607 21393 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 793649a9666f4e89aa24efe161541746 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 17:57:13.536845 21393 raft_consensus.cc:686] T 00000000000000000000000000000000 P 793649a9666f4e89aa24efe161541746 [term 1 LEADER]: Becoming Leader. State: Replica: 793649a9666f4e89aa24efe161541746, State: Running, Role: LEADER
I20260929 17:57:13.537215 21393 consensus_queue.cc:237] T 00000000000000000000000000000000 P 793649a9666f4e89aa24efe161541746 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "793649a9666f4e89aa24efe161541746" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43073 } }
I20260929 17:57:13.537899 21385 sys_catalog.cc:564] T 00000000000000000000000000000000 P 793649a9666f4e89aa24efe161541746 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 17:57:13.538414 21393 sys_catalog.cc:455] T 00000000000000000000000000000000 P 793649a9666f4e89aa24efe161541746 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 793649a9666f4e89aa24efe161541746. Latest consensus state: current_term: 1 leader_uuid: "793649a9666f4e89aa24efe161541746" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "793649a9666f4e89aa24efe161541746" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43073 } } }
I20260929 17:57:13.538532 21393 sys_catalog.cc:458] T 00000000000000000000000000000000 P 793649a9666f4e89aa24efe161541746 [sys.catalog]: This master's current role is: LEADER
I20260929 17:57:13.538935 21395 sys_catalog.cc:455] T 00000000000000000000000000000000 P 793649a9666f4e89aa24efe161541746 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "793649a9666f4e89aa24efe161541746" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "793649a9666f4e89aa24efe161541746" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43073 } } }
I20260929 17:57:13.539026 21395 sys_catalog.cc:458] T 00000000000000000000000000000000 P 793649a9666f4e89aa24efe161541746 [sys.catalog]: This master's current role is: LEADER
W20260929 17:57:13.542951 21410 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 793649a9666f4e89aa24efe161541746: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 17:57:13.543032 21410 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 17:57:13.543094 21401 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 17:57:13.543767 21401 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 17:57:13.548149 21388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:13.537319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.108:54759 (local address 127.0.0.1:43073)
0929 17:57:13.537543 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0929 17:57:13.537548 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:57:13.540358 (+ 2810us) server_negotiation.cc:407] Connection header received
0929 17:57:13.540477 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:57:13.540480 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:57:13.540523 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:57:13.540596 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:57:13.541323 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:13.541838 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:13.542634 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:13.542866 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:13.542954 (+ 88us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:13.544305 (+ 1351us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:57:13.544308 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:57:13.544310 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:57:13.544334 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:57:13.545791 (+ 1457us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:57:13.546396 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:57:13.546399 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:57:13.546401 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:57:13.546497 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:57:13.546845 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:57:13.546848 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:57:13.546850 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:57:13.547091 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:57:13.547233 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0929 17:57:13.547945 (+ 712us) server_negotiation.cc:299] Negotiation successful
0929 17:57:13.548026 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144}
I20260929 17:57:13.550009 21401 catalog_manager.cc:1347] Generated new cluster ID: d48b170ded9748e38d7bc083124b7a2b
I20260929 17:57:13.550076 21401 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 17:57:13.560432 21401 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 17:57:13.561710 21401 catalog_manager.cc:1504] Loading token signing keys...
I20260929 17:57:13.574617 21401 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 793649a9666f4e89aa24efe161541746: Generated new TSK 0
I20260929 17:57:13.575410 21401 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 17:57:13.586251 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskup3Soe/build/debug/bin/kudu with pid 21317
W20260929 17:57:13.591042 21302 connection.cc:535] client connection to 127.0.0.1:43073 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43073): Connection reset by peer (error 104)
2026-09-29T17:57:13Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1336 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790704375975685-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 17:57:13 dist-test-slave-53nh krb5kdc[21416](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 17:57:13 dist-test-slave-53nh krb5kdc[21416](info): set up 2 sockets
Sep 29 17:57:13 dist-test-slave-53nh krb5kdc[21416](info): commencing operation
krb5kdc: starting...
W20260929 17:57:14.533155 392 mini_kdc.cc:121] Time spent starting KDC: real 0.928s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790704375975685-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790704375975685-392-0/krb5kdc/test-user.keytab.
Sep 29 17:57:14 dist-test-slave-53nh krb5kdc[21416](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704634, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T17:57:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T17:57:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790704375975685-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790704375975685-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 17:57:14.655294 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskup3Soe/build/debug/bin/kudu
/tmp/dist-test-taskup3Soe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790704375975685-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790704375975685-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790704375975685-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790704375975685-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35919
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790704375975685-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35919
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790704375975685-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790704375975685-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790704375975685-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:57:14.764940 21429 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:57:14.765288 21429 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:57:14.765359 21429 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:57:14.769475 21429 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 17:57:14.769551 21429 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:57:14.769578 21429 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:57:14.769605 21429 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 17:57:14.769631 21429 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 17:57:14.775163 21429 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790704375975685-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790704375975685-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35919
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790704375975685-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35919
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790704375975685-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21429
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790704375975685-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:57:14.776312 21429 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:57:14.777486 21429 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:57:14.786258 21437 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:57:14.786270 21435 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:57:14.786270 21434 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:57:14.786624 21429 server_base.cc:1034] running on GCE node
I20260929 17:57:14.787230 21429 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:57:14.788367 21429 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:57:14.789525 21429 hybrid_clock.cc:648] HybridClock initialized: now 1790704634789506 us; error 32 us; skew 500 ppm
Sep 29 17:57:14 dist-test-slave-53nh krb5kdc[21416](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704634, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:57:14.797511 21429 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 17:57:14.798480 21429 webserver.cc:460] Webserver started at http://127.0.0.1:39175/ using document root <none> and password file <none>
I20260929 17:57:14.799038 21429 fs_manager.cc:359] Metadata directory not provided
I20260929 17:57:14.799101 21429 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:57:14.799283 21429 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:57:14.801403 21429 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790704375975685-392-0/minicluster-data/master-0/data/instance:
uuid: "7341270886024988833fffff5965c4a5"
format_stamp: "Formatted at 2026-09-29 17:57:14 on dist-test-slave-53nh"
server_key: "330103304d5708acbdf7a6b3a3777c9c"
server_key_iv: "9468da31d5b705d9ef574982973fbcf3"
server_key_version: "encryptionkey@0"
I20260929 17:57:14.801927 21429 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790704375975685-392-0/minicluster-data/master-0/wal/instance:
uuid: "7341270886024988833fffff5965c4a5"
format_stamp: "Formatted at 2026-09-29 17:57:14 on dist-test-slave-53nh"
server_key: "330103304d5708acbdf7a6b3a3777c9c"
server_key_iv: "9468da31d5b705d9ef574982973fbcf3"
server_key_version: "encryptionkey@0"
I20260929 17:57:14.805646 21429 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260929 17:57:14.808085 21444 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:57:14.809352 21429 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 17:57:14.809501 21429 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790704375975685-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790704375975685-392-0/minicluster-data/master-0/wal
uuid: "7341270886024988833fffff5965c4a5"
format_stamp: "Formatted at 2026-09-29 17:57:14 on dist-test-slave-53nh"
server_key: "330103304d5708acbdf7a6b3a3777c9c"
server_key_iv: "9468da31d5b705d9ef574982973fbcf3"
server_key_version: "encryptionkey@0"
I20260929 17:57:14.809609 21429 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790704375975685-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790704375975685-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790704375975685-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:57:14.821758 21429 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:57:14.825038 21429 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:57:14.825227 21429 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:57:14.832355 21429 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35919
I20260929 17:57:14.832365 21496 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35919 every 8 connection(s)
I20260929 17:57:14.833345 21429 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790704375975685-392-0/minicluster-data/master-0/data/info.pb
I20260929 17:57:14.836135 21497 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:57:14.841779 21497 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7341270886024988833fffff5965c4a5: Bootstrap starting.
I20260929 17:57:14.841831 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskup3Soe/build/debug/bin/kudu as pid 21429
I20260929 17:57:14.841941 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790704375975685-392-0/minicluster-data/master-0/wal/instance
I20260929 17:57:14.842217 392 external_mini_cluster.cc:1428] Setting key 192b291a677d228697dd8c99895d56b6
I20260929 17:57:14.844408 21497 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7341270886024988833fffff5965c4a5: Neither blocks nor log segments found. Creating new log.
I20260929 17:57:14.845168 21497 log.cc:824] T 00000000000000000000000000000000 P 7341270886024988833fffff5965c4a5: Log is configured to *not* fsync() on all Append() calls
I20260929 17:57:14.847107 21497 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7341270886024988833fffff5965c4a5: No bootstrap required, opened a new log
I20260929 17:57:14.851192 21500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:14.843537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.108:41199 (local address 127.0.0.1:35919)
0929 17:57:14.843935 (+ 398us) server_negotiation.cc:206] Beginning negotiation
0929 17:57:14.843959 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0929 17:57:14.844000 (+ 41us) server_negotiation.cc:407] Connection header received
0929 17:57:14.844699 (+ 699us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:57:14.844719 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:57:14.845124 (+ 405us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:57:14.845435 (+ 311us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:57:14.845852 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:14.846830 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:14.848007 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:14.848486 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:14.848614 (+ 128us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:14.849487 (+ 873us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:57:14.849492 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:57:14.849505 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 17:57:14.849887 (+ 382us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:57:14.849963 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 17:57:14.849986 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:57:14.850087 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0929 17:57:14.850264 (+ 177us) server_negotiation.cc:299] Negotiation successful
0929 17:57:14.850401 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":259,"thread_start_us":110,"threads_started":1}
I20260929 17:57:14.851081 21497 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7341270886024988833fffff5965c4a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7341270886024988833fffff5965c4a5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35919 } }
I20260929 17:57:14.851285 21497 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7341270886024988833fffff5965c4a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:57:14.851336 21497 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7341270886024988833fffff5965c4a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7341270886024988833fffff5965c4a5, State: Initialized, Role: FOLLOWER
I20260929 17:57:14.851761 21497 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7341270886024988833fffff5965c4a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7341270886024988833fffff5965c4a5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35919 } }
I20260929 17:57:14.851891 21497 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7341270886024988833fffff5965c4a5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 17:57:14.851953 21497 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7341270886024988833fffff5965c4a5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 17:57:14.852039 21497 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7341270886024988833fffff5965c4a5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:57:14.852905 21497 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7341270886024988833fffff5965c4a5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7341270886024988833fffff5965c4a5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35919 } }
I20260929 17:57:14.853262 21497 leader_election.cc:302] T 00000000000000000000000000000000 P 7341270886024988833fffff5965c4a5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7341270886024988833fffff5965c4a5; no voters:
I20260929 17:57:14.853576 21497 leader_election.cc:288] T 00000000000000000000000000000000 P 7341270886024988833fffff5965c4a5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 17:57:14.853688 21502 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7341270886024988833fffff5965c4a5 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 17:57:14.853896 21502 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7341270886024988833fffff5965c4a5 [term 1 LEADER]: Becoming Leader. State: Replica: 7341270886024988833fffff5965c4a5, State: Running, Role: LEADER
I20260929 17:57:14.854054 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 17:57:14.854259 21502 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7341270886024988833fffff5965c4a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7341270886024988833fffff5965c4a5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35919 } }
I20260929 17:57:14.854874 21497 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7341270886024988833fffff5965c4a5 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 17:57:14.855684 21504 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7341270886024988833fffff5965c4a5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7341270886024988833fffff5965c4a5. Latest consensus state: current_term: 1 leader_uuid: "7341270886024988833fffff5965c4a5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7341270886024988833fffff5965c4a5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35919 } } }
I20260929 17:57:14.855794 21504 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7341270886024988833fffff5965c4a5 [sys.catalog]: This master's current role is: LEADER
I20260929 17:57:14.856123 21503 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7341270886024988833fffff5965c4a5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7341270886024988833fffff5965c4a5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7341270886024988833fffff5965c4a5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35919 } } }
I20260929 17:57:14.856217 21503 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7341270886024988833fffff5965c4a5 [sys.catalog]: This master's current role is: LEADER
I20260929 17:57:14.856488 21515 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 17:57:14.859912 21515 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 17:57:14.862504 21500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:14.857171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.108:39077 (local address 127.0.0.1:35919)
0929 17:57:14.857278 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0929 17:57:14.857283 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:57:14.857719 (+ 436us) server_negotiation.cc:407] Connection header received
0929 17:57:14.857871 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:57:14.857875 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:57:14.857929 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:57:14.857999 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:57:14.858352 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:14.859117 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:14.860301 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:14.860615 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:14.860688 (+ 73us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:14.862007 (+ 1319us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:57:14.862011 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:57:14.862015 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 17:57:14.862029 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:57:14.862060 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 17:57:14.862074 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:57:14.862168 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0929 17:57:14.862312 (+ 144us) server_negotiation.cc:299] Negotiation successful
0929 17:57:14.862352 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260929 17:57:14.865975 21515 catalog_manager.cc:1347] Generated new cluster ID: 01854aaa54da474782d248d014d6f174
I20260929 17:57:14.866055 21515 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 17:57:14.883625 21515 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 17:57:14.884588 21515 catalog_manager.cc:1504] Loading token signing keys...
I20260929 17:57:14.897185 21515 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7341270886024988833fffff5965c4a5: Generated new TSK 0
I20260929 17:57:14.898201 21515 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 17:57:14.909906 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskup3Soe/build/debug/bin/kudu with pid 21429
W20260929 17:57:14.914541 21414 connection.cc:535] client connection to 127.0.0.1:35919 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35919): Connection reset by peer (error 104)
2026-09-29T17:57:14Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1323 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790704375975685-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 17:57:14 dist-test-slave-53nh krb5kdc[21528](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 17:57:14 dist-test-slave-53nh krb5kdc[21528](info): set up 2 sockets
Sep 29 17:57:14 dist-test-slave-53nh krb5kdc[21528](info): commencing operation
krb5kdc: starting...
W20260929 17:57:15.865187 392 mini_kdc.cc:121] Time spent starting KDC: real 0.937s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790704375975685-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790704375975685-392-0/krb5kdc/test-user.keytab.
Sep 29 17:57:15 dist-test-slave-53nh krb5kdc[21528](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704635, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T17:57:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T17:57:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790704375975685-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790704375975685-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 17:57:15.984056 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskup3Soe/build/debug/bin/kudu
/tmp/dist-test-taskup3Soe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790704375975685-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790704375975685-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790704375975685-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790704375975685-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36709
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790704375975685-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36709
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790704375975685-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790704375975685-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790704375975685-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:57:16.093338 21541 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:57:16.093634 21541 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:57:16.093695 21541 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:57:16.097841 21541 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 17:57:16.097926 21541 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:57:16.097952 21541 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:57:16.097972 21541 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 17:57:16.097993 21541 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 17:57:16.103710 21541 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790704375975685-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790704375975685-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36709
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790704375975685-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36709
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790704375975685-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21541
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790704375975685-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:57:16.104835 21541 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:57:16.105887 21541 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:57:16.114370 21549 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:57:16.114375 21547 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:57:16.114370 21546 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:57:16.114984 21541 server_base.cc:1034] running on GCE node
I20260929 17:57:16.115628 21541 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:57:16.116813 21541 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:57:16.117971 21541 hybrid_clock.cc:648] HybridClock initialized: now 1790704636117955 us; error 37 us; skew 500 ppm
Sep 29 17:57:16 dist-test-slave-53nh krb5kdc[21528](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704636, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:57:16.120697 21541 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 17:57:16.121667 21541 webserver.cc:460] Webserver started at http://127.0.0.1:40101/ using document root <none> and password file <none>
I20260929 17:57:16.122241 21541 fs_manager.cc:359] Metadata directory not provided
I20260929 17:57:16.122315 21541 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:57:16.122534 21541 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:57:16.124629 21541 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790704375975685-392-0/minicluster-data/master-0/data/instance:
uuid: "4e5d0392a0db4f66bccecd789b153a13"
format_stamp: "Formatted at 2026-09-29 17:57:16 on dist-test-slave-53nh"
server_key: "6e6ea6c2a5e63be35d3ba519f42697b9"
server_key_iv: "c404c2318ff93db14b7556e7162b8c96"
server_key_version: "encryptionkey@0"
I20260929 17:57:16.125200 21541 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790704375975685-392-0/minicluster-data/master-0/wal/instance:
uuid: "4e5d0392a0db4f66bccecd789b153a13"
format_stamp: "Formatted at 2026-09-29 17:57:16 on dist-test-slave-53nh"
server_key: "6e6ea6c2a5e63be35d3ba519f42697b9"
server_key_iv: "c404c2318ff93db14b7556e7162b8c96"
server_key_version: "encryptionkey@0"
I20260929 17:57:16.129030 21541 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260929 17:57:16.131433 21556 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:57:16.132694 21541 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 17:57:16.132812 21541 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790704375975685-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790704375975685-392-0/minicluster-data/master-0/wal
uuid: "4e5d0392a0db4f66bccecd789b153a13"
format_stamp: "Formatted at 2026-09-29 17:57:16 on dist-test-slave-53nh"
server_key: "6e6ea6c2a5e63be35d3ba519f42697b9"
server_key_iv: "c404c2318ff93db14b7556e7162b8c96"
server_key_version: "encryptionkey@0"
I20260929 17:57:16.132915 21541 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790704375975685-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790704375975685-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790704375975685-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:57:16.137759 21541 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:57:16.141247 21541 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:57:16.141434 21541 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:57:16.148396 21541 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36709
I20260929 17:57:16.148473 21608 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36709 every 8 connection(s)
I20260929 17:57:16.149451 21541 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790704375975685-392-0/minicluster-data/master-0/data/info.pb
I20260929 17:57:16.150134 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskup3Soe/build/debug/bin/kudu as pid 21541
I20260929 17:57:16.150261 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790704375975685-392-0/minicluster-data/master-0/wal/instance
I20260929 17:57:16.150548 392 external_mini_cluster.cc:1428] Setting key 44448ce88fcc11c977118f33de0cbd93
I20260929 17:57:16.152817 21609 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:57:16.155735 21612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:16.151924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.108:40773 (local address 127.0.0.1:36709)
0929 17:57:16.152371 (+ 447us) server_negotiation.cc:206] Beginning negotiation
0929 17:57:16.152391 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0929 17:57:16.152428 (+ 37us) server_negotiation.cc:407] Connection header received
0929 17:57:16.153149 (+ 721us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:57:16.153168 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:57:16.153462 (+ 294us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:57:16.153786 (+ 324us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:57:16.154554 (+ 768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:57:16.154558 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:57:16.154570 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 17:57:16.154580 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:57:16.154632 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 17:57:16.154651 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:57:16.154776 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0929 17:57:16.154871 (+ 95us) server_negotiation.cc:299] Negotiation successful
0929 17:57:16.155031 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294,"thread_start_us":131,"threads_started":1}
I20260929 17:57:16.158474 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 17:57:16.158705 21609 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e5d0392a0db4f66bccecd789b153a13: Bootstrap starting.
I20260929 17:57:16.161046 21609 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4e5d0392a0db4f66bccecd789b153a13: Neither blocks nor log segments found. Creating new log.
I20260929 17:57:16.161885 21612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:16.160385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.108:44355 (local address 127.0.0.1:36709)
0929 17:57:16.160505 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0929 17:57:16.160510 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:57:16.160679 (+ 169us) server_negotiation.cc:407] Connection header received
0929 17:57:16.160761 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:57:16.160765 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:57:16.160802 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:57:16.160883 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:57:16.161538 (+ 655us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:57:16.161542 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:57:16.161545 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 17:57:16.161556 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:57:16.161585 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 17:57:16.161597 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:57:16.161679 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0929 17:57:16.161728 (+ 49us) server_negotiation.cc:299] Negotiation successful
0929 17:57:16.161766 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 17:57:16.161914 21609 log.cc:824] T 00000000000000000000000000000000 P 4e5d0392a0db4f66bccecd789b153a13: Log is configured to *not* fsync() on all Append() calls
I20260929 17:57:16.163789 21609 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e5d0392a0db4f66bccecd789b153a13: No bootstrap required, opened a new log
I20260929 17:57:16.167521 21609 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4e5d0392a0db4f66bccecd789b153a13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e5d0392a0db4f66bccecd789b153a13" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36709 } }
I20260929 17:57:16.167763 21609 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4e5d0392a0db4f66bccecd789b153a13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:57:16.167797 21609 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4e5d0392a0db4f66bccecd789b153a13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e5d0392a0db4f66bccecd789b153a13, State: Initialized, Role: FOLLOWER
I20260929 17:57:16.168175 21609 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4e5d0392a0db4f66bccecd789b153a13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e5d0392a0db4f66bccecd789b153a13" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36709 } }
I20260929 17:57:16.168291 21609 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4e5d0392a0db4f66bccecd789b153a13 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 17:57:16.168336 21609 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4e5d0392a0db4f66bccecd789b153a13 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 17:57:16.168406 21609 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4e5d0392a0db4f66bccecd789b153a13 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:57:16.169363 21609 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4e5d0392a0db4f66bccecd789b153a13 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e5d0392a0db4f66bccecd789b153a13" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36709 } }
I20260929 17:57:16.169690 21609 leader_election.cc:302] T 00000000000000000000000000000000 P 4e5d0392a0db4f66bccecd789b153a13 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4e5d0392a0db4f66bccecd789b153a13; no voters:
I20260929 17:57:16.169991 21609 leader_election.cc:288] T 00000000000000000000000000000000 P 4e5d0392a0db4f66bccecd789b153a13 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 17:57:16.170114 21619 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4e5d0392a0db4f66bccecd789b153a13 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 17:57:16.170380 21619 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4e5d0392a0db4f66bccecd789b153a13 [term 1 LEADER]: Becoming Leader. State: Replica: 4e5d0392a0db4f66bccecd789b153a13, State: Running, Role: LEADER
I20260929 17:57:16.170712 21619 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4e5d0392a0db4f66bccecd789b153a13 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e5d0392a0db4f66bccecd789b153a13" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36709 } }
I20260929 17:57:16.171275 21609 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4e5d0392a0db4f66bccecd789b153a13 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 17:57:16.172442 21621 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e5d0392a0db4f66bccecd789b153a13 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4e5d0392a0db4f66bccecd789b153a13. Latest consensus state: current_term: 1 leader_uuid: "4e5d0392a0db4f66bccecd789b153a13" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e5d0392a0db4f66bccecd789b153a13" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36709 } } }
I20260929 17:57:16.172502 21620 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e5d0392a0db4f66bccecd789b153a13 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4e5d0392a0db4f66bccecd789b153a13" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e5d0392a0db4f66bccecd789b153a13" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36709 } } }
I20260929 17:57:16.172549 21621 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e5d0392a0db4f66bccecd789b153a13 [sys.catalog]: This master's current role is: LEADER
I20260929 17:57:16.172874 21620 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e5d0392a0db4f66bccecd789b153a13 [sys.catalog]: This master's current role is: LEADER
I20260929 17:57:16.173022 21628 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 17:57:16.176426 21628 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 17:57:16.185576 21628 catalog_manager.cc:1347] Generated new cluster ID: 0bf5d4fd14834161b414167a7e7b9555
I20260929 17:57:16.185668 21628 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 17:57:16.192294 21628 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 17:57:16.192683 21628 catalog_manager.cc:1504] Loading token signing keys...
I20260929 17:57:16.210052 21628 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4e5d0392a0db4f66bccecd789b153a13: Generated new TSK 0
I20260929 17:57:16.210821 21628 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 17:57:16.214915 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskup3Soe/build/debug/bin/kudu with pid 21541
2026-09-29T17:57:16Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1305 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790704375975685-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 17:57:16 dist-test-slave-53nh krb5kdc[21640](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 17:57:16 dist-test-slave-53nh krb5kdc[21640](info): set up 2 sockets
Sep 29 17:57:16 dist-test-slave-53nh krb5kdc[21640](info): commencing operation
krb5kdc: starting...
W20260929 17:57:17.172924 392 mini_kdc.cc:121] Time spent starting KDC: real 0.939s user 0.004s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790704375975685-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790704375975685-392-0/krb5kdc/test-user.keytab.
Sep 29 17:57:17 dist-test-slave-53nh krb5kdc[21640](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704637, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T17:57:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T17:57:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790704375975685-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790704375975685-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 17:57:17.292577 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskup3Soe/build/debug/bin/kudu
/tmp/dist-test-taskup3Soe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790704375975685-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790704375975685-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790704375975685-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790704375975685-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42973
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790704375975685-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42973
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790704375975685-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790704375975685-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790704375975685-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:57:17.404219 21653 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:57:17.404502 21653 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:57:17.404543 21653 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:57:17.408577 21653 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 17:57:17.408644 21653 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:57:17.408663 21653 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:57:17.408677 21653 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 17:57:17.408691 21653 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 17:57:17.414142 21653 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790704375975685-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790704375975685-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42973
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790704375975685-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42973
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790704375975685-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21653
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790704375975685-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:57:17.415179 21653 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:57:17.416155 21653 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:57:17.425596 21659 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:57:17.425611 21661 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:57:17.425578 21658 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:57:17.426017 21653 server_base.cc:1034] running on GCE node
I20260929 17:57:17.426575 21653 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:57:17.427742 21653 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:57:17.428885 21653 hybrid_clock.cc:648] HybridClock initialized: now 1790704637428868 us; error 33 us; skew 500 ppm
Sep 29 17:57:17 dist-test-slave-53nh krb5kdc[21640](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704637, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:57:17.431771 21653 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 17:57:17.432711 21653 webserver.cc:460] Webserver started at http://127.0.0.1:36965/ using document root <none> and password file <none>
I20260929 17:57:17.433295 21653 fs_manager.cc:359] Metadata directory not provided
I20260929 17:57:17.433358 21653 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:57:17.433535 21653 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:57:17.435547 21653 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790704375975685-392-0/minicluster-data/master-0/data/instance:
uuid: "c274b23fc44f4a4584c1bc9d8e35dd60"
format_stamp: "Formatted at 2026-09-29 17:57:17 on dist-test-slave-53nh"
server_key: "2f8cd2ffaf6dfe8e6501418daad0f32c"
server_key_iv: "ef1c79ac3802fbfd40087191e7ee05c6"
server_key_version: "encryptionkey@0"
I20260929 17:57:17.436062 21653 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790704375975685-392-0/minicluster-data/master-0/wal/instance:
uuid: "c274b23fc44f4a4584c1bc9d8e35dd60"
format_stamp: "Formatted at 2026-09-29 17:57:17 on dist-test-slave-53nh"
server_key: "2f8cd2ffaf6dfe8e6501418daad0f32c"
server_key_iv: "ef1c79ac3802fbfd40087191e7ee05c6"
server_key_version: "encryptionkey@0"
I20260929 17:57:17.439661 21653 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260929 17:57:17.442061 21668 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:57:17.443249 21653 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 17:57:17.443354 21653 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790704375975685-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790704375975685-392-0/minicluster-data/master-0/wal
uuid: "c274b23fc44f4a4584c1bc9d8e35dd60"
format_stamp: "Formatted at 2026-09-29 17:57:17 on dist-test-slave-53nh"
server_key: "2f8cd2ffaf6dfe8e6501418daad0f32c"
server_key_iv: "ef1c79ac3802fbfd40087191e7ee05c6"
server_key_version: "encryptionkey@0"
I20260929 17:57:17.443459 21653 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790704375975685-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790704375975685-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790704375975685-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:57:17.468019 21653 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:57:17.471132 21653 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:57:17.471310 21653 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:57:17.478197 21653 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42973
I20260929 17:57:17.478221 21720 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42973 every 8 connection(s)
I20260929 17:57:17.479133 21653 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790704375975685-392-0/minicluster-data/master-0/data/info.pb
I20260929 17:57:17.479779 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskup3Soe/build/debug/bin/kudu as pid 21653
I20260929 17:57:17.479879 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790704375975685-392-0/minicluster-data/master-0/wal/instance
I20260929 17:57:17.480140 392 external_mini_cluster.cc:1428] Setting key 05a6f8d58547d4a44f2b6ba780fad906
I20260929 17:57:17.482290 21721 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:57:17.488745 21721 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c274b23fc44f4a4584c1bc9d8e35dd60: Bootstrap starting.
Sep 29 17:57:17 dist-test-slave-53nh krb5kdc[21640](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704637, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260929 17:57:17.491012 21721 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c274b23fc44f4a4584c1bc9d8e35dd60: Neither blocks nor log segments found. Creating new log.
I20260929 17:57:17.491707 21721 log.cc:824] T 00000000000000000000000000000000 P c274b23fc44f4a4584c1bc9d8e35dd60: Log is configured to *not* fsync() on all Append() calls
I20260929 17:57:17.494071 21721 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c274b23fc44f4a4584c1bc9d8e35dd60: No bootstrap required, opened a new log
I20260929 17:57:17.494993 21724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:17.481649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.108:40153 (local address 127.0.0.1:42973)
0929 17:57:17.482076 (+ 427us) server_negotiation.cc:206] Beginning negotiation
0929 17:57:17.482091 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 17:57:17.482127 (+ 36us) server_negotiation.cc:407] Connection header received
0929 17:57:17.482796 (+ 669us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:57:17.482814 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:57:17.483174 (+ 360us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:57:17.483463 (+ 289us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:57:17.484313 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:17.485359 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:17.486508 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:17.487048 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:17.487193 (+ 145us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:17.489965 (+ 2772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:57:17.489970 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:57:17.489984 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:57:17.490012 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:57:17.491604 (+ 1592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:57:17.492118 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:57:17.492123 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:57:17.492128 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:57:17.492233 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:57:17.492614 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:57:17.492618 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:57:17.492620 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:57:17.493275 (+ 655us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:57:17.493483 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0929 17:57:17.493770 (+ 287us) server_negotiation.cc:299] Negotiation successful
0929 17:57:17.494046 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":291,"thread_start_us":118,"threads_started":1}
I20260929 17:57:17.497994 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 17:57:17.498196 21721 raft_consensus.cc:348] T 00000000000000000000000000000000 P c274b23fc44f4a4584c1bc9d8e35dd60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c274b23fc44f4a4584c1bc9d8e35dd60" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42973 } }
I20260929 17:57:17.498440 21721 raft_consensus.cc:374] T 00000000000000000000000000000000 P c274b23fc44f4a4584c1bc9d8e35dd60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:57:17.498505 21721 raft_consensus.cc:729] T 00000000000000000000000000000000 P c274b23fc44f4a4584c1bc9d8e35dd60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c274b23fc44f4a4584c1bc9d8e35dd60, State: Initialized, Role: FOLLOWER
I20260929 17:57:17.498942 21721 consensus_queue.cc:260] T 00000000000000000000000000000000 P c274b23fc44f4a4584c1bc9d8e35dd60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c274b23fc44f4a4584c1bc9d8e35dd60" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42973 } }
I20260929 17:57:17.499073 21721 raft_consensus.cc:388] T 00000000000000000000000000000000 P c274b23fc44f4a4584c1bc9d8e35dd60 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 17:57:17.499167 21721 raft_consensus.cc:482] T 00000000000000000000000000000000 P c274b23fc44f4a4584c1bc9d8e35dd60 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 17:57:17.499277 21721 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c274b23fc44f4a4584c1bc9d8e35dd60 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:57:17.500420 21721 raft_consensus.cc:504] T 00000000000000000000000000000000 P c274b23fc44f4a4584c1bc9d8e35dd60 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c274b23fc44f4a4584c1bc9d8e35dd60" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42973 } }
I20260929 17:57:17.500818 21721 leader_election.cc:302] T 00000000000000000000000000000000 P c274b23fc44f4a4584c1bc9d8e35dd60 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c274b23fc44f4a4584c1bc9d8e35dd60; no voters:
I20260929 17:57:17.501163 21721 leader_election.cc:288] T 00000000000000000000000000000000 P c274b23fc44f4a4584c1bc9d8e35dd60 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 17:57:17.501269 21731 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c274b23fc44f4a4584c1bc9d8e35dd60 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 17:57:17.501482 21731 raft_consensus.cc:686] T 00000000000000000000000000000000 P c274b23fc44f4a4584c1bc9d8e35dd60 [term 1 LEADER]: Becoming Leader. State: Replica: c274b23fc44f4a4584c1bc9d8e35dd60, State: Running, Role: LEADER
I20260929 17:57:17.501840 21731 consensus_queue.cc:237] T 00000000000000000000000000000000 P c274b23fc44f4a4584c1bc9d8e35dd60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c274b23fc44f4a4584c1bc9d8e35dd60" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42973 } }
I20260929 17:57:17.502521 21721 sys_catalog.cc:564] T 00000000000000000000000000000000 P c274b23fc44f4a4584c1bc9d8e35dd60 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 17:57:17.503104 21731 sys_catalog.cc:455] T 00000000000000000000000000000000 P c274b23fc44f4a4584c1bc9d8e35dd60 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c274b23fc44f4a4584c1bc9d8e35dd60. Latest consensus state: current_term: 1 leader_uuid: "c274b23fc44f4a4584c1bc9d8e35dd60" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c274b23fc44f4a4584c1bc9d8e35dd60" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42973 } } }
I20260929 17:57:17.503278 21731 sys_catalog.cc:458] T 00000000000000000000000000000000 P c274b23fc44f4a4584c1bc9d8e35dd60 [sys.catalog]: This master's current role is: LEADER
I20260929 17:57:17.503684 21740 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 17:57:17.505288 21732 sys_catalog.cc:455] T 00000000000000000000000000000000 P c274b23fc44f4a4584c1bc9d8e35dd60 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c274b23fc44f4a4584c1bc9d8e35dd60" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c274b23fc44f4a4584c1bc9d8e35dd60" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42973 } } }
I20260929 17:57:17.505388 21732 sys_catalog.cc:458] T 00000000000000000000000000000000 P c274b23fc44f4a4584c1bc9d8e35dd60 [sys.catalog]: This master's current role is: LEADER
I20260929 17:57:17.507560 21740 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 17:57:17.508944 21724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:17.500064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.108:58899 (local address 127.0.0.1:42973)
0929 17:57:17.500192 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0929 17:57:17.500196 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:57:17.500293 (+ 97us) server_negotiation.cc:407] Connection header received
0929 17:57:17.500388 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:57:17.500391 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:57:17.500448 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:57:17.500534 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:57:17.501311 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:17.502095 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:17.503241 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:17.503566 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:17.503661 (+ 95us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:17.505449 (+ 1788us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:57:17.505453 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:57:17.505455 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:57:17.505483 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:57:17.507074 (+ 1591us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:57:17.507534 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:57:17.507537 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:57:17.507539 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:57:17.507683 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:57:17.508112 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:57:17.508115 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:57:17.508117 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:57:17.508304 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:57:17.508430 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0929 17:57:17.508709 (+ 279us) server_negotiation.cc:299] Negotiation successful
0929 17:57:17.508800 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 17:57:17.513911 21740 catalog_manager.cc:1347] Generated new cluster ID: 36625d167911407aa7d8ce33dffa0fdc
I20260929 17:57:17.513975 21740 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 17:57:17.541509 21740 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 17:57:17.542526 21740 catalog_manager.cc:1504] Loading token signing keys...
I20260929 17:57:17.551156 21740 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c274b23fc44f4a4584c1bc9d8e35dd60: Generated new TSK 0
I20260929 17:57:17.551910 21740 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 17:57:17.567194 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskup3Soe/build/debug/bin/kudu with pid 21653
W20260929 17:57:17.572083 21638 connection.cc:535] client connection to 127.0.0.1:42973 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42973): Connection reset by peer (error 104)
2026-09-29T17:57:17Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1352 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790704375975685-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 17:57:17 dist-test-slave-53nh krb5kdc[21752](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 17:57:17 dist-test-slave-53nh krb5kdc[21752](info): set up 2 sockets
Sep 29 17:57:17 dist-test-slave-53nh krb5kdc[21752](info): commencing operation
krb5kdc: starting...
W20260929 17:57:18.513315 392 mini_kdc.cc:121] Time spent starting KDC: real 0.927s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790704375975685-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790704375975685-392-0/krb5kdc/test-user.keytab.
Sep 29 17:57:18 dist-test-slave-53nh krb5kdc[21752](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704638, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T17:57:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T17:57:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790704375975685-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790704375975685-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 17:57:18.630165 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskup3Soe/build/debug/bin/kudu
/tmp/dist-test-taskup3Soe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790704375975685-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790704375975685-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790704375975685-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790704375975685-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35711
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790704375975685-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35711
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790704375975685-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790704375975685-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790704375975685-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:57:18.743399 21765 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:57:18.743739 21765 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:57:18.743803 21765 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:57:18.748167 21765 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 17:57:18.748257 21765 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:57:18.748286 21765 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:57:18.748307 21765 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 17:57:18.748328 21765 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 17:57:18.754135 21765 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790704375975685-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790704375975685-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35711
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790704375975685-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35711
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790704375975685-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21765
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790704375975685-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:57:18.755275 21765 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:57:18.756354 21765 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:57:18.765182 21770 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:57:18.765182 21773 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:57:18.765182 21771 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:57:18.765694 21765 server_base.cc:1034] running on GCE node
I20260929 17:57:18.766232 21765 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:57:18.767376 21765 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:57:18.768533 21765 hybrid_clock.cc:648] HybridClock initialized: now 1790704638768514 us; error 32 us; skew 500 ppm
Sep 29 17:57:18 dist-test-slave-53nh krb5kdc[21752](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704638, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:57:18.771239 21765 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 17:57:18.772172 21765 webserver.cc:460] Webserver started at http://127.0.0.1:45077/ using document root <none> and password file <none>
I20260929 17:57:18.772716 21765 fs_manager.cc:359] Metadata directory not provided
I20260929 17:57:18.772789 21765 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:57:18.772994 21765 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:57:18.775107 21765 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790704375975685-392-0/minicluster-data/master-0/data/instance:
uuid: "9a76607d9adc4bfca1c4149a78fa9433"
format_stamp: "Formatted at 2026-09-29 17:57:18 on dist-test-slave-53nh"
server_key: "de48de65dde6ef2a75b1db4c0f9f436e"
server_key_iv: "fd07e58770c8a90a00b965c13c72b824"
server_key_version: "encryptionkey@0"
I20260929 17:57:18.775651 21765 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790704375975685-392-0/minicluster-data/master-0/wal/instance:
uuid: "9a76607d9adc4bfca1c4149a78fa9433"
format_stamp: "Formatted at 2026-09-29 17:57:18 on dist-test-slave-53nh"
server_key: "de48de65dde6ef2a75b1db4c0f9f436e"
server_key_iv: "fd07e58770c8a90a00b965c13c72b824"
server_key_version: "encryptionkey@0"
I20260929 17:57:18.779415 21765 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 17:57:18.782058 21780 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:57:18.783360 21765 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 17:57:18.783483 21765 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790704375975685-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790704375975685-392-0/minicluster-data/master-0/wal
uuid: "9a76607d9adc4bfca1c4149a78fa9433"
format_stamp: "Formatted at 2026-09-29 17:57:18 on dist-test-slave-53nh"
server_key: "de48de65dde6ef2a75b1db4c0f9f436e"
server_key_iv: "fd07e58770c8a90a00b965c13c72b824"
server_key_version: "encryptionkey@0"
I20260929 17:57:18.783591 21765 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790704375975685-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790704375975685-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790704375975685-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:57:18.804226 21765 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:57:18.807065 21765 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:57:18.807263 21765 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:57:18.814010 21765 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35711
I20260929 17:57:18.814028 21832 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35711 every 8 connection(s)
I20260929 17:57:18.814953 21765 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790704375975685-392-0/minicluster-data/master-0/data/info.pb
I20260929 17:57:18.816695 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskup3Soe/build/debug/bin/kudu as pid 21765
I20260929 17:57:18.816829 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790704375975685-392-0/minicluster-data/master-0/wal/instance
I20260929 17:57:18.817148 392 external_mini_cluster.cc:1428] Setting key f462f44ff7ccc5005f9bf16625b56944
I20260929 17:57:18.818024 21833 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:57:18.824066 21833 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a76607d9adc4bfca1c4149a78fa9433: Bootstrap starting.
I20260929 17:57:18.826402 21836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:18.818546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.108:57249 (local address 127.0.0.1:35711)
0929 17:57:18.819013 (+ 467us) server_negotiation.cc:206] Beginning negotiation
0929 17:57:18.819038 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0929 17:57:18.819075 (+ 37us) server_negotiation.cc:407] Connection header received
0929 17:57:18.819736 (+ 661us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:57:18.819753 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:57:18.820126 (+ 373us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:57:18.820466 (+ 340us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:57:18.820918 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:18.822003 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:18.822930 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:18.823440 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:18.823587 (+ 147us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:18.824609 (+ 1022us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:57:18.824615 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:57:18.824629 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 17:57:18.825092 (+ 463us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:57:18.825151 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 17:57:18.825175 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:57:18.825293 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0929 17:57:18.825486 (+ 193us) server_negotiation.cc:299] Negotiation successful
0929 17:57:18.825644 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":147,"threads_started":1}
I20260929 17:57:18.826519 21833 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9a76607d9adc4bfca1c4149a78fa9433: Neither blocks nor log segments found. Creating new log.
I20260929 17:57:18.827420 21833 log.cc:824] T 00000000000000000000000000000000 P 9a76607d9adc4bfca1c4149a78fa9433: Log is configured to *not* fsync() on all Append() calls
I20260929 17:57:18.829497 21833 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a76607d9adc4bfca1c4149a78fa9433: No bootstrap required, opened a new log
I20260929 17:57:18.829543 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 17:57:18.833588 21833 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9a76607d9adc4bfca1c4149a78fa9433 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a76607d9adc4bfca1c4149a78fa9433" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35711 } }
I20260929 17:57:18.833861 21833 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9a76607d9adc4bfca1c4149a78fa9433 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:57:18.833920 21833 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9a76607d9adc4bfca1c4149a78fa9433 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a76607d9adc4bfca1c4149a78fa9433, State: Initialized, Role: FOLLOWER
I20260929 17:57:18.834426 21833 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9a76607d9adc4bfca1c4149a78fa9433 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a76607d9adc4bfca1c4149a78fa9433" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35711 } }
I20260929 17:57:18.834584 21833 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9a76607d9adc4bfca1c4149a78fa9433 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 17:57:18.834652 21833 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9a76607d9adc4bfca1c4149a78fa9433 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 17:57:18.834750 21833 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9a76607d9adc4bfca1c4149a78fa9433 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:57:18.835433 21836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:18.831496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.108:34683 (local address 127.0.0.1:35711)
0929 17:57:18.831625 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0929 17:57:18.831637 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 17:57:18.831726 (+ 89us) server_negotiation.cc:407] Connection header received
0929 17:57:18.831819 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:57:18.831822 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:57:18.831869 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:57:18.831942 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:57:18.832158 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:18.832924 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:18.833753 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:18.834089 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:18.834204 (+ 115us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:18.835041 (+ 837us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:57:18.835044 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:57:18.835048 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 17:57:18.835062 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:57:18.835090 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 17:57:18.835103 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:57:18.835183 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0929 17:57:18.835268 (+ 85us) server_negotiation.cc:299] Negotiation successful
0929 17:57:18.835305 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 17:57:18.835851 21833 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9a76607d9adc4bfca1c4149a78fa9433 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a76607d9adc4bfca1c4149a78fa9433" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35711 } }
I20260929 17:57:18.836195 21833 leader_election.cc:302] T 00000000000000000000000000000000 P 9a76607d9adc4bfca1c4149a78fa9433 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9a76607d9adc4bfca1c4149a78fa9433; no voters:
I20260929 17:57:18.836521 21833 leader_election.cc:288] T 00000000000000000000000000000000 P 9a76607d9adc4bfca1c4149a78fa9433 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 17:57:18.836791 21843 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9a76607d9adc4bfca1c4149a78fa9433 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 17:57:18.837036 21843 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9a76607d9adc4bfca1c4149a78fa9433 [term 1 LEADER]: Becoming Leader. State: Replica: 9a76607d9adc4bfca1c4149a78fa9433, State: Running, Role: LEADER
I20260929 17:57:18.837399 21843 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9a76607d9adc4bfca1c4149a78fa9433 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a76607d9adc4bfca1c4149a78fa9433" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35711 } }
I20260929 17:57:18.837884 21833 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9a76607d9adc4bfca1c4149a78fa9433 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 17:57:18.839051 21844 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a76607d9adc4bfca1c4149a78fa9433 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9a76607d9adc4bfca1c4149a78fa9433" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a76607d9adc4bfca1c4149a78fa9433" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35711 } } }
I20260929 17:57:18.839097 21845 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a76607d9adc4bfca1c4149a78fa9433 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9a76607d9adc4bfca1c4149a78fa9433. Latest consensus state: current_term: 1 leader_uuid: "9a76607d9adc4bfca1c4149a78fa9433" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a76607d9adc4bfca1c4149a78fa9433" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35711 } } }
I20260929 17:57:18.839157 21844 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a76607d9adc4bfca1c4149a78fa9433 [sys.catalog]: This master's current role is: LEADER
I20260929 17:57:18.839250 21845 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a76607d9adc4bfca1c4149a78fa9433 [sys.catalog]: This master's current role is: LEADER
I20260929 17:57:18.839622 21854 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 17:57:18.842854 21854 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 17:57:18.847997 21854 catalog_manager.cc:1347] Generated new cluster ID: de8618fa1ab944888457331d3b81a295
I20260929 17:57:18.848071 21854 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 17:57:18.861222 21854 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 17:57:18.862782 21854 catalog_manager.cc:1504] Loading token signing keys...
I20260929 17:57:18.878522 21854 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9a76607d9adc4bfca1c4149a78fa9433: Generated new TSK 0
I20260929 17:57:18.879441 21854 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 17:57:18.887441 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskup3Soe/build/debug/bin/kudu with pid 21765
W20260929 17:57:18.892076 21750 connection.cc:535] client connection to 127.0.0.1:35711 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35711): Connection reset by peer (error 104)
2026-09-29T17:57:18Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1320 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790704375975685-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 17:57:18 dist-test-slave-53nh krb5kdc[21864](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 17:57:18 dist-test-slave-53nh krb5kdc[21864](info): set up 2 sockets
Sep 29 17:57:18 dist-test-slave-53nh krb5kdc[21864](info): commencing operation
krb5kdc: starting...
W20260929 17:57:19.837211 392 mini_kdc.cc:121] Time spent starting KDC: real 0.931s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790704375975685-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790704375975685-392-0/krb5kdc/test-user.keytab.
Sep 29 17:57:19 dist-test-slave-53nh krb5kdc[21864](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704639, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T17:57:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T17:57:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790704375975685-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790704375975685-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 17:57:19.956562 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskup3Soe/build/debug/bin/kudu
/tmp/dist-test-taskup3Soe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790704375975685-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790704375975685-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790704375975685-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790704375975685-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41643
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790704375975685-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44255
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41643
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790704375975685-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790704375975685-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790704375975685-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:57:20.068696 21877 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:57:20.068986 21877 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:57:20.069118 21877 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:57:20.073318 21877 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 17:57:20.073388 21877 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:57:20.073414 21877 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:57:20.073446 21877 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 17:57:20.073475 21877 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 17:57:20.079000 21877 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44255
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790704375975685-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790704375975685-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41643
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790704375975685-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41643
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790704375975685-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21877
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790704375975685-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:57:20.080072 21877 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:57:20.081112 21877 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:57:20.089498 21883 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:57:20.089533 21882 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:57:20.089586 21877 server_base.cc:1034] running on GCE node
W20260929 17:57:20.089531 21885 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:57:20.090305 21877 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:57:20.091441 21877 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:57:20.092624 21877 hybrid_clock.cc:648] HybridClock initialized: now 1790704640092602 us; error 34 us; skew 500 ppm
Sep 29 17:57:20 dist-test-slave-53nh krb5kdc[21864](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704640, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:57:20.098762 21877 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 17:57:20.099785 21877 webserver.cc:460] Webserver started at http://127.0.0.1:43909/ using document root <none> and password file <none>
I20260929 17:57:20.100401 21877 fs_manager.cc:359] Metadata directory not provided
I20260929 17:57:20.100477 21877 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:57:20.100682 21877 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:57:20.103036 21877 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790704375975685-392-0/minicluster-data/master-0/data/instance:
uuid: "bdd41741bb8b450988788176b11294fc"
format_stamp: "Formatted at 2026-09-29 17:57:20 on dist-test-slave-53nh"
server_key: "6c04b710c82a68a435ce5d9f0e623ca2"
server_key_iv: "ee2bccbeab4d67186030dd648de9169d"
server_key_version: "encryptionkey@0"
I20260929 17:57:20.103622 21877 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790704375975685-392-0/minicluster-data/master-0/wal/instance:
uuid: "bdd41741bb8b450988788176b11294fc"
format_stamp: "Formatted at 2026-09-29 17:57:20 on dist-test-slave-53nh"
server_key: "6c04b710c82a68a435ce5d9f0e623ca2"
server_key_iv: "ee2bccbeab4d67186030dd648de9169d"
server_key_version: "encryptionkey@0"
I20260929 17:57:20.107795 21877 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 17:57:20.110670 21892 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:57:20.112190 21877 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 17:57:20.112385 21877 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790704375975685-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790704375975685-392-0/minicluster-data/master-0/wal
uuid: "bdd41741bb8b450988788176b11294fc"
format_stamp: "Formatted at 2026-09-29 17:57:20 on dist-test-slave-53nh"
server_key: "6c04b710c82a68a435ce5d9f0e623ca2"
server_key_iv: "ee2bccbeab4d67186030dd648de9169d"
server_key_version: "encryptionkey@0"
I20260929 17:57:20.112511 21877 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790704375975685-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790704375975685-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790704375975685-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:57:20.117777 21877 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:57:20.121063 21877 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:57:20.121241 21877 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:57:20.128340 21877 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41643
I20260929 17:57:20.128350 21944 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41643 every 8 connection(s)
I20260929 17:57:20.129339 21877 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790704375975685-392-0/minicluster-data/master-0/data/info.pb
I20260929 17:57:20.132056 21945 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:57:20.132474 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskup3Soe/build/debug/bin/kudu as pid 21877
I20260929 17:57:20.132582 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790704375975685-392-0/minicluster-data/master-0/wal/instance
I20260929 17:57:20.132871 392 external_mini_cluster.cc:1428] Setting key 462e9d3ae200428e1fe477b524481688
I20260929 17:57:20.137951 21948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:20.134245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.108:57041 (local address 127.0.0.1:41643)
0929 17:57:20.134658 (+ 413us) server_negotiation.cc:206] Beginning negotiation
0929 17:57:20.134673 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 17:57:20.134708 (+ 35us) server_negotiation.cc:407] Connection header received
0929 17:57:20.135368 (+ 660us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:57:20.135385 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:57:20.135675 (+ 290us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:57:20.135990 (+ 315us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:57:20.136755 (+ 765us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:57:20.136761 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:57:20.136774 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 17:57:20.136784 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:57:20.136835 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 17:57:20.136853 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:57:20.136973 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0929 17:57:20.137090 (+ 117us) server_negotiation.cc:299] Negotiation successful
0929 17:57:20.137262 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":130,"threads_started":1}
I20260929 17:57:20.138773 21945 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bdd41741bb8b450988788176b11294fc: Bootstrap starting.
I20260929 17:57:20.141103 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 17:57:20.141587 21945 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bdd41741bb8b450988788176b11294fc: Neither blocks nor log segments found. Creating new log.
I20260929 17:57:20.142553 21945 log.cc:824] T 00000000000000000000000000000000 P bdd41741bb8b450988788176b11294fc: Log is configured to *not* fsync() on all Append() calls
I20260929 17:57:20.144878 21945 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bdd41741bb8b450988788176b11294fc: No bootstrap required, opened a new log
I20260929 17:57:20.145413 21948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:20.143515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.108:36513 (local address 127.0.0.1:41643)
0929 17:57:20.143641 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0929 17:57:20.143646 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:57:20.143824 (+ 178us) server_negotiation.cc:407] Connection header received
0929 17:57:20.143922 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:57:20.143925 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:57:20.143961 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:57:20.144050 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:57:20.144848 (+ 798us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:57:20.144852 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:57:20.144854 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 17:57:20.144866 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:57:20.144897 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 17:57:20.144911 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:57:20.145025 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0929 17:57:20.145249 (+ 224us) server_negotiation.cc:299] Negotiation successful
0929 17:57:20.145295 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260929 17:57:20.148787 21945 raft_consensus.cc:348] T 00000000000000000000000000000000 P bdd41741bb8b450988788176b11294fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bdd41741bb8b450988788176b11294fc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41643 } }
I20260929 17:57:20.149086 21945 raft_consensus.cc:374] T 00000000000000000000000000000000 P bdd41741bb8b450988788176b11294fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:57:20.149154 21945 raft_consensus.cc:729] T 00000000000000000000000000000000 P bdd41741bb8b450988788176b11294fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bdd41741bb8b450988788176b11294fc, State: Initialized, Role: FOLLOWER
I20260929 17:57:20.149571 21945 consensus_queue.cc:260] T 00000000000000000000000000000000 P bdd41741bb8b450988788176b11294fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bdd41741bb8b450988788176b11294fc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41643 } }
I20260929 17:57:20.149693 21945 raft_consensus.cc:388] T 00000000000000000000000000000000 P bdd41741bb8b450988788176b11294fc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 17:57:20.149734 21945 raft_consensus.cc:482] T 00000000000000000000000000000000 P bdd41741bb8b450988788176b11294fc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 17:57:20.149816 21945 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bdd41741bb8b450988788176b11294fc [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:57:20.150908 21945 raft_consensus.cc:504] T 00000000000000000000000000000000 P bdd41741bb8b450988788176b11294fc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bdd41741bb8b450988788176b11294fc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41643 } }
I20260929 17:57:20.151263 21945 leader_election.cc:302] T 00000000000000000000000000000000 P bdd41741bb8b450988788176b11294fc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bdd41741bb8b450988788176b11294fc; no voters:
I20260929 17:57:20.151569 21945 leader_election.cc:288] T 00000000000000000000000000000000 P bdd41741bb8b450988788176b11294fc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 17:57:20.151726 21955 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bdd41741bb8b450988788176b11294fc [term 1 FOLLOWER]: Leader election won for term 1
I20260929 17:57:20.151935 21955 raft_consensus.cc:686] T 00000000000000000000000000000000 P bdd41741bb8b450988788176b11294fc [term 1 LEADER]: Becoming Leader. State: Replica: bdd41741bb8b450988788176b11294fc, State: Running, Role: LEADER
I20260929 17:57:20.152281 21955 consensus_queue.cc:237] T 00000000000000000000000000000000 P bdd41741bb8b450988788176b11294fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bdd41741bb8b450988788176b11294fc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41643 } }
I20260929 17:57:20.152798 21945 sys_catalog.cc:564] T 00000000000000000000000000000000 P bdd41741bb8b450988788176b11294fc [sys.catalog]: configured and running, proceeding with master startup.
I20260929 17:57:20.153950 21957 sys_catalog.cc:455] T 00000000000000000000000000000000 P bdd41741bb8b450988788176b11294fc [sys.catalog]: SysCatalogTable state changed. Reason: New leader bdd41741bb8b450988788176b11294fc. Latest consensus state: current_term: 1 leader_uuid: "bdd41741bb8b450988788176b11294fc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bdd41741bb8b450988788176b11294fc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41643 } } }
I20260929 17:57:20.154078 21957 sys_catalog.cc:458] T 00000000000000000000000000000000 P bdd41741bb8b450988788176b11294fc [sys.catalog]: This master's current role is: LEADER
I20260929 17:57:20.154264 21956 sys_catalog.cc:455] T 00000000000000000000000000000000 P bdd41741bb8b450988788176b11294fc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bdd41741bb8b450988788176b11294fc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bdd41741bb8b450988788176b11294fc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41643 } } }
I20260929 17:57:20.154392 21956 sys_catalog.cc:458] T 00000000000000000000000000000000 P bdd41741bb8b450988788176b11294fc [sys.catalog]: This master's current role is: LEADER
I20260929 17:57:20.154662 21964 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 17:57:20.157526 21964 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 17:57:20.163189 21964 catalog_manager.cc:1347] Generated new cluster ID: a5dcb38ce3fa45388ab4fa1ab60ed24f
I20260929 17:57:20.163264 21964 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 17:57:20.181133 21964 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 17:57:20.181509 21964 catalog_manager.cc:1504] Loading token signing keys...
I20260929 17:57:20.190158 21964 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bdd41741bb8b450988788176b11294fc: Generated new TSK 0
I20260929 17:57:20.190918 21964 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 17:57:20.196914 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskup3Soe/build/debug/bin/kudu with pid 21877
2026-09-29T17:57:20Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1309 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790704375975685-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 17:57:20 dist-test-slave-53nh krb5kdc[21976](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 17:57:20 dist-test-slave-53nh krb5kdc[21976](info): set up 2 sockets
Sep 29 17:57:20 dist-test-slave-53nh krb5kdc[21976](info): commencing operation
krb5kdc: starting...
W20260929 17:57:21.161051 392 mini_kdc.cc:121] Time spent starting KDC: real 0.945s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790704375975685-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790704375975685-392-0/krb5kdc/test-user.keytab.
Sep 29 17:57:21 dist-test-slave-53nh krb5kdc[21976](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704641, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T17:57:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T17:57:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790704375975685-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790704375975685-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 17:57:21.281232 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskup3Soe/build/debug/bin/kudu
/tmp/dist-test-taskup3Soe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790704375975685-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790704375975685-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790704375975685-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790704375975685-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36365
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790704375975685-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36365
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790704375975685-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790704375975685-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790704375975685-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:57:21.392961 21989 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:57:21.393282 21989 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:57:21.393345 21989 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:57:21.397517 21989 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 17:57:21.397617 21989 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:57:21.397648 21989 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:57:21.397671 21989 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 17:57:21.397701 21989 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 17:57:21.403155 21989 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790704375975685-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790704375975685-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36365
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790704375975685-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36365
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790704375975685-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21989
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790704375975685-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:57:21.404227 21989 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:57:21.405279 21989 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 17:57:21.413972 21989 server_base.cc:1034] running on GCE node
W20260929 17:57:21.413905 21997 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:57:21.413987 21994 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:57:21.413905 21995 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:57:21.414700 21989 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:57:21.415834 21989 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:57:21.417040 21989 hybrid_clock.cc:648] HybridClock initialized: now 1790704641417017 us; error 34 us; skew 500 ppm
Sep 29 17:57:21 dist-test-slave-53nh krb5kdc[21976](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704641, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:57:21.419994 21989 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 17:57:21.421105 21989 webserver.cc:460] Webserver started at http://127.0.98.62:39357/ using document root <none> and password file <none>
I20260929 17:57:21.421711 21989 fs_manager.cc:359] Metadata directory not provided
I20260929 17:57:21.421787 21989 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:57:21.421998 21989 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:57:21.424186 21989 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790704375975685-392-0/minicluster-data/master-0/data/instance:
uuid: "0fe50599be4c496bb98e2f660b00ba45"
format_stamp: "Formatted at 2026-09-29 17:57:21 on dist-test-slave-53nh"
server_key: "e601112c9496bc79be44942f80b7d729"
server_key_iv: "7729c1101ba051b3fd4e0f1e80b4691d"
server_key_version: "encryptionkey@0"
I20260929 17:57:21.424772 21989 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790704375975685-392-0/minicluster-data/master-0/wal/instance:
uuid: "0fe50599be4c496bb98e2f660b00ba45"
format_stamp: "Formatted at 2026-09-29 17:57:21 on dist-test-slave-53nh"
server_key: "e601112c9496bc79be44942f80b7d729"
server_key_iv: "7729c1101ba051b3fd4e0f1e80b4691d"
server_key_version: "encryptionkey@0"
I20260929 17:57:21.428983 21989 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260929 17:57:21.431733 22004 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:57:21.432983 21989 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260929 17:57:21.433157 21989 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790704375975685-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790704375975685-392-0/minicluster-data/master-0/wal
uuid: "0fe50599be4c496bb98e2f660b00ba45"
format_stamp: "Formatted at 2026-09-29 17:57:21 on dist-test-slave-53nh"
server_key: "e601112c9496bc79be44942f80b7d729"
server_key_iv: "7729c1101ba051b3fd4e0f1e80b4691d"
server_key_version: "encryptionkey@0"
I20260929 17:57:21.433291 21989 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790704375975685-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790704375975685-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790704375975685-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:57:21.451112 21989 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:57:21.454767 21989 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:57:21.454984 21989 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:57:21.462267 21989 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36365
I20260929 17:57:21.462285 22056 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36365 every 8 connection(s)
I20260929 17:57:21.463338 21989 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790704375975685-392-0/minicluster-data/master-0/data/info.pb
I20260929 17:57:21.466465 22057 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:57:21.467645 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskup3Soe/build/debug/bin/kudu as pid 21989
I20260929 17:57:21.467772 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790704375975685-392-0/minicluster-data/master-0/wal/instance
I20260929 17:57:21.468034 392 external_mini_cluster.cc:1428] Setting key cc2b3b06bebc9653946ebe05aa9dfd03
I20260929 17:57:21.473078 22057 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0fe50599be4c496bb98e2f660b00ba45: Bootstrap starting.
I20260929 17:57:21.475805 22057 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0fe50599be4c496bb98e2f660b00ba45: Neither blocks nor log segments found. Creating new log.
I20260929 17:57:21.476727 22057 log.cc:824] T 00000000000000000000000000000000 P 0fe50599be4c496bb98e2f660b00ba45: Log is configured to *not* fsync() on all Append() calls
Sep 29 17:57:21 dist-test-slave-53nh krb5kdc[21976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704641, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 17:57:21.479207 22057 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0fe50599be4c496bb98e2f660b00ba45: No bootstrap required, opened a new log
I20260929 17:57:21.483057 22060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:21.469594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58653 (local address 127.0.98.62:36365)
0929 17:57:21.470027 (+ 433us) server_negotiation.cc:206] Beginning negotiation
0929 17:57:21.470043 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 17:57:21.470082 (+ 39us) server_negotiation.cc:407] Connection header received
0929 17:57:21.470735 (+ 653us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:57:21.470754 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:57:21.471130 (+ 376us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:57:21.471572 (+ 442us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:57:21.472445 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:21.473627 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:21.474806 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:21.475303 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:21.475428 (+ 125us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:21.478363 (+ 2935us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:57:21.478369 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:57:21.478384 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:57:21.478412 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:57:21.480119 (+ 1707us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:57:21.480646 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:57:21.480652 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:57:21.480656 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:57:21.480782 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:57:21.481113 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:57:21.481116 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:57:21.481118 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:57:21.481523 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:57:21.481744 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0929 17:57:21.482043 (+ 299us) server_negotiation.cc:299] Negotiation successful
0929 17:57:21.482267 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":287,"thread_start_us":132,"threads_started":1}
I20260929 17:57:21.482945 22057 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0fe50599be4c496bb98e2f660b00ba45 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fe50599be4c496bb98e2f660b00ba45" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36365 } }
I20260929 17:57:21.483191 22057 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0fe50599be4c496bb98e2f660b00ba45 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:57:21.483249 22057 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0fe50599be4c496bb98e2f660b00ba45 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fe50599be4c496bb98e2f660b00ba45, State: Initialized, Role: FOLLOWER
I20260929 17:57:21.483703 22057 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0fe50599be4c496bb98e2f660b00ba45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fe50599be4c496bb98e2f660b00ba45" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36365 } }
I20260929 17:57:21.483842 22057 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0fe50599be4c496bb98e2f660b00ba45 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 17:57:21.483948 22057 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0fe50599be4c496bb98e2f660b00ba45 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 17:57:21.484058 22057 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0fe50599be4c496bb98e2f660b00ba45 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:57:21.485041 22057 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0fe50599be4c496bb98e2f660b00ba45 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fe50599be4c496bb98e2f660b00ba45" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36365 } }
I20260929 17:57:21.485378 22057 leader_election.cc:302] T 00000000000000000000000000000000 P 0fe50599be4c496bb98e2f660b00ba45 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0fe50599be4c496bb98e2f660b00ba45; no voters:
I20260929 17:57:21.485708 22057 leader_election.cc:288] T 00000000000000000000000000000000 P 0fe50599be4c496bb98e2f660b00ba45 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 17:57:21.485833 22062 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0fe50599be4c496bb98e2f660b00ba45 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 17:57:21.486066 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 17:57:21.486055 22062 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0fe50599be4c496bb98e2f660b00ba45 [term 1 LEADER]: Becoming Leader. State: Replica: 0fe50599be4c496bb98e2f660b00ba45, State: Running, Role: LEADER
I20260929 17:57:21.486353 22062 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0fe50599be4c496bb98e2f660b00ba45 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fe50599be4c496bb98e2f660b00ba45" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36365 } }
I20260929 17:57:21.486869 22057 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0fe50599be4c496bb98e2f660b00ba45 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 17:57:21.488010 22063 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0fe50599be4c496bb98e2f660b00ba45 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0fe50599be4c496bb98e2f660b00ba45" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fe50599be4c496bb98e2f660b00ba45" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36365 } } }
I20260929 17:57:21.488149 22063 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0fe50599be4c496bb98e2f660b00ba45 [sys.catalog]: This master's current role is: LEADER
I20260929 17:57:21.488440 22064 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0fe50599be4c496bb98e2f660b00ba45 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0fe50599be4c496bb98e2f660b00ba45. Latest consensus state: current_term: 1 leader_uuid: "0fe50599be4c496bb98e2f660b00ba45" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fe50599be4c496bb98e2f660b00ba45" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36365 } } }
I20260929 17:57:21.488546 22064 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0fe50599be4c496bb98e2f660b00ba45 [sys.catalog]: This master's current role is: LEADER
W20260929 17:57:21.491734 22082 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0fe50599be4c496bb98e2f660b00ba45: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 17:57:21.491817 22082 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 17:57:21.491876 22076 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 17:57:21.492682 22076 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 17:57:21.499782 22060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:21.488286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56431 (local address 127.0.98.62:36365)
0929 17:57:21.488674 (+ 388us) server_negotiation.cc:206] Beginning negotiation
0929 17:57:21.488678 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:57:21.488693 (+ 15us) server_negotiation.cc:407] Connection header received
0929 17:57:21.488733 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:57:21.488736 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:57:21.488783 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:57:21.488858 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:57:21.489896 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:21.490646 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:21.491748 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:21.492082 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:21.492194 (+ 112us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:21.494103 (+ 1909us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:57:21.494107 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:57:21.494110 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:57:21.494139 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:57:21.497665 (+ 3526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:57:21.498313 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:57:21.498317 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:57:21.498319 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:57:21.498409 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:57:21.498810 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:57:21.498814 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:57:21.498816 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:57:21.499044 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:57:21.499191 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0929 17:57:21.499508 (+ 317us) server_negotiation.cc:299] Negotiation successful
0929 17:57:21.499613 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":321}
I20260929 17:57:21.500164 22076 catalog_manager.cc:1347] Generated new cluster ID: 2541120f06f245e586dd086c4ddd0a0a
I20260929 17:57:21.500217 22076 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 17:57:21.512302 22076 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 17:57:21.513481 22076 catalog_manager.cc:1504] Loading token signing keys...
I20260929 17:57:21.520934 22076 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0fe50599be4c496bb98e2f660b00ba45: Generated new TSK 0
I20260929 17:57:21.521705 22076 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 17:57:21.527629 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskup3Soe/build/debug/bin/kudu with pid 21989
W20260929 17:57:21.532442 21974 connection.cc:535] client connection to 127.0.98.62:36365 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:36365): Connection reset by peer (error 104)
2026-09-29T17:57:21Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1331 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790704375975685-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 17:57:21 dist-test-slave-53nh krb5kdc[22088](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 17:57:21 dist-test-slave-53nh krb5kdc[22088](info): set up 2 sockets
Sep 29 17:57:21 dist-test-slave-53nh krb5kdc[22088](info): commencing operation
krb5kdc: starting...
W20260929 17:57:22.481261 392 mini_kdc.cc:121] Time spent starting KDC: real 0.934s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790704375975685-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790704375975685-392-0/krb5kdc/test-user.keytab.
Sep 29 17:57:22 dist-test-slave-53nh krb5kdc[22088](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704642, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T17:57:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T17:57:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790704375975685-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790704375975685-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 17:57:22.597760 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskup3Soe/build/debug/bin/kudu
/tmp/dist-test-taskup3Soe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790704375975685-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790704375975685-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790704375975685-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790704375975685-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46753
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790704375975685-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46753
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790704375975685-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790704375975685-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790704375975685-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:57:22.705262 22101 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:57:22.705559 22101 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:57:22.705601 22101 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:57:22.709676 22101 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 17:57:22.709745 22101 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:57:22.709764 22101 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:57:22.709779 22101 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 17:57:22.709793 22101 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 17:57:22.715226 22101 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790704375975685-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790704375975685-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46753
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790704375975685-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46753
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790704375975685-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22101
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790704375975685-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:57:22.716328 22101 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:57:22.717326 22101 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:57:22.725432 22106 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:57:22.725425 22107 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:57:22.725616 22101 server_base.cc:1034] running on GCE node
W20260929 17:57:22.725425 22109 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:57:22.726219 22101 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:57:22.727270 22101 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:57:22.728423 22101 hybrid_clock.cc:648] HybridClock initialized: now 1790704642728399 us; error 38 us; skew 500 ppm
Sep 29 17:57:22 dist-test-slave-53nh krb5kdc[22088](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704642, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:57:22.731037 22101 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 17:57:22.731959 22101 webserver.cc:460] Webserver started at http://127.0.98.62:34505/ using document root <none> and password file <none>
I20260929 17:57:22.732496 22101 fs_manager.cc:359] Metadata directory not provided
I20260929 17:57:22.732560 22101 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:57:22.732758 22101 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:57:22.734807 22101 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790704375975685-392-0/minicluster-data/master-0/data/instance:
uuid: "0c7bbaf563e14b5bb9186eafcac163d1"
format_stamp: "Formatted at 2026-09-29 17:57:22 on dist-test-slave-53nh"
server_key: "039c9225d6b25ab8eb1d9f2e1969e707"
server_key_iv: "dbcaa0013c920e3ab0498920e6c81f3e"
server_key_version: "encryptionkey@0"
I20260929 17:57:22.735347 22101 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790704375975685-392-0/minicluster-data/master-0/wal/instance:
uuid: "0c7bbaf563e14b5bb9186eafcac163d1"
format_stamp: "Formatted at 2026-09-29 17:57:22 on dist-test-slave-53nh"
server_key: "039c9225d6b25ab8eb1d9f2e1969e707"
server_key_iv: "dbcaa0013c920e3ab0498920e6c81f3e"
server_key_version: "encryptionkey@0"
I20260929 17:57:22.738962 22101 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 17:57:22.741379 22116 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:57:22.742617 22101 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 17:57:22.742748 22101 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790704375975685-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790704375975685-392-0/minicluster-data/master-0/wal
uuid: "0c7bbaf563e14b5bb9186eafcac163d1"
format_stamp: "Formatted at 2026-09-29 17:57:22 on dist-test-slave-53nh"
server_key: "039c9225d6b25ab8eb1d9f2e1969e707"
server_key_iv: "dbcaa0013c920e3ab0498920e6c81f3e"
server_key_version: "encryptionkey@0"
I20260929 17:57:22.742864 22101 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790704375975685-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790704375975685-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790704375975685-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:57:22.756489 22101 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:57:22.759469 22101 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:57:22.759748 22101 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:57:22.766507 22101 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46753
I20260929 17:57:22.766520 22168 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46753 every 8 connection(s)
I20260929 17:57:22.767441 22101 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790704375975685-392-0/minicluster-data/master-0/data/info.pb
I20260929 17:57:22.770201 22169 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:57:22.773705 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskup3Soe/build/debug/bin/kudu as pid 22101
I20260929 17:57:22.773825 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790704375975685-392-0/minicluster-data/master-0/wal/instance
I20260929 17:57:22.774070 392 external_mini_cluster.cc:1428] Setting key 29b6b80ffc987092c137b5043343cd2d
I20260929 17:57:22.775785 22169 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c7bbaf563e14b5bb9186eafcac163d1: Bootstrap starting.
I20260929 17:57:22.778024 22169 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0c7bbaf563e14b5bb9186eafcac163d1: Neither blocks nor log segments found. Creating new log.
I20260929 17:57:22.778791 22169 log.cc:824] T 00000000000000000000000000000000 P 0c7bbaf563e14b5bb9186eafcac163d1: Log is configured to *not* fsync() on all Append() calls
I20260929 17:57:22.780874 22169 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c7bbaf563e14b5bb9186eafcac163d1: No bootstrap required, opened a new log
I20260929 17:57:22.783005 22172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:22.775354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36697 (local address 127.0.98.62:46753)
0929 17:57:22.775754 (+ 400us) server_negotiation.cc:206] Beginning negotiation
0929 17:57:22.775770 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 17:57:22.775806 (+ 36us) server_negotiation.cc:407] Connection header received
0929 17:57:22.776438 (+ 632us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:57:22.776455 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:57:22.776797 (+ 342us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:57:22.777097 (+ 300us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:57:22.777449 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:22.778426 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:22.779519 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:22.779987 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:22.780102 (+ 115us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:22.781082 (+ 980us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:57:22.781087 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:57:22.781100 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 17:57:22.781493 (+ 393us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:57:22.781548 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 17:57:22.781571 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:57:22.781681 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0929 17:57:22.781854 (+ 173us) server_negotiation.cc:299] Negotiation successful
0929 17:57:22.781986 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":260,"thread_start_us":117,"threads_started":1}
I20260929 17:57:22.784035 22169 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0c7bbaf563e14b5bb9186eafcac163d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c7bbaf563e14b5bb9186eafcac163d1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46753 } }
I20260929 17:57:22.784267 22169 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0c7bbaf563e14b5bb9186eafcac163d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:57:22.784315 22169 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0c7bbaf563e14b5bb9186eafcac163d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c7bbaf563e14b5bb9186eafcac163d1, State: Initialized, Role: FOLLOWER
I20260929 17:57:22.784731 22169 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0c7bbaf563e14b5bb9186eafcac163d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c7bbaf563e14b5bb9186eafcac163d1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46753 } }
I20260929 17:57:22.784861 22169 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0c7bbaf563e14b5bb9186eafcac163d1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 17:57:22.784969 22169 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0c7bbaf563e14b5bb9186eafcac163d1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 17:57:22.785091 22169 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0c7bbaf563e14b5bb9186eafcac163d1 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:57:22.785974 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 17:57:22.786249 22169 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0c7bbaf563e14b5bb9186eafcac163d1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c7bbaf563e14b5bb9186eafcac163d1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46753 } }
I20260929 17:57:22.786630 22169 leader_election.cc:302] T 00000000000000000000000000000000 P 0c7bbaf563e14b5bb9186eafcac163d1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0c7bbaf563e14b5bb9186eafcac163d1; no voters:
I20260929 17:57:22.786970 22169 leader_election.cc:288] T 00000000000000000000000000000000 P 0c7bbaf563e14b5bb9186eafcac163d1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 17:57:22.787077 22174 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0c7bbaf563e14b5bb9186eafcac163d1 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 17:57:22.787302 22174 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0c7bbaf563e14b5bb9186eafcac163d1 [term 1 LEADER]: Becoming Leader. State: Replica: 0c7bbaf563e14b5bb9186eafcac163d1, State: Running, Role: LEADER
I20260929 17:57:22.787640 22174 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0c7bbaf563e14b5bb9186eafcac163d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c7bbaf563e14b5bb9186eafcac163d1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46753 } }
I20260929 17:57:22.788270 22169 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0c7bbaf563e14b5bb9186eafcac163d1 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 17:57:22.789194 22175 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c7bbaf563e14b5bb9186eafcac163d1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0c7bbaf563e14b5bb9186eafcac163d1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c7bbaf563e14b5bb9186eafcac163d1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46753 } } }
I20260929 17:57:22.789316 22175 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c7bbaf563e14b5bb9186eafcac163d1 [sys.catalog]: This master's current role is: LEADER
I20260929 17:57:22.789520 22178 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c7bbaf563e14b5bb9186eafcac163d1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0c7bbaf563e14b5bb9186eafcac163d1. Latest consensus state: current_term: 1 leader_uuid: "0c7bbaf563e14b5bb9186eafcac163d1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c7bbaf563e14b5bb9186eafcac163d1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46753 } } }
I20260929 17:57:22.789660 22178 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c7bbaf563e14b5bb9186eafcac163d1 [sys.catalog]: This master's current role is: LEADER
I20260929 17:57:22.789919 22188 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 17:57:22.793121 22188 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 17:57:22.794209 22172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:22.789059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39119 (local address 127.0.98.62:46753)
0929 17:57:22.789153 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0929 17:57:22.789157 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:57:22.789612 (+ 455us) server_negotiation.cc:407] Connection header received
0929 17:57:22.789805 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:57:22.789809 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:57:22.789859 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:57:22.789914 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:57:22.790350 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:22.791079 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:22.792181 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:22.792490 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:22.792589 (+ 99us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:22.793629 (+ 1040us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:57:22.793633 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:57:22.793636 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 17:57:22.793651 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:57:22.793683 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 17:57:22.793697 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:57:22.793763 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0929 17:57:22.794032 (+ 269us) server_negotiation.cc:299] Negotiation successful
0929 17:57:22.794069 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260929 17:57:22.799034 22188 catalog_manager.cc:1347] Generated new cluster ID: e861bce6e12f43e391e28539352cb352
I20260929 17:57:22.799108 22188 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 17:57:22.812498 22188 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 17:57:22.813501 22188 catalog_manager.cc:1504] Loading token signing keys...
I20260929 17:57:22.830336 22188 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0c7bbaf563e14b5bb9186eafcac163d1: Generated new TSK 0
I20260929 17:57:22.831110 22188 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 17:57:22.841650 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskup3Soe/build/debug/bin/kudu with pid 22101
W20260929 17:57:22.846517 22086 connection.cc:535] client connection to 127.0.98.62:46753 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:46753): Connection reset by peer (error 104)
2026-09-29T17:57:22Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1313 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790704375975685-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 17:57:22 dist-test-slave-53nh krb5kdc[22200](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 17:57:22 dist-test-slave-53nh krb5kdc[22200](info): set up 2 sockets
Sep 29 17:57:22 dist-test-slave-53nh krb5kdc[22200](info): commencing operation
krb5kdc: starting...
W20260929 17:57:23.797106 392 mini_kdc.cc:121] Time spent starting KDC: real 0.937s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790704375975685-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790704375975685-392-0/krb5kdc/test-user.keytab.
Sep 29 17:57:23 dist-test-slave-53nh krb5kdc[22200](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704643, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T17:57:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T17:57:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790704375975685-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790704375975685-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 17:57:23.916639 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskup3Soe/build/debug/bin/kudu
/tmp/dist-test-taskup3Soe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790704375975685-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790704375975685-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790704375975685-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790704375975685-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40201
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790704375975685-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40201
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790704375975685-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790704375975685-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790704375975685-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:57:24.027956 22213 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:57:24.028259 22213 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:57:24.028321 22213 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:57:24.032614 22213 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 17:57:24.032709 22213 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:57:24.032739 22213 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:57:24.032760 22213 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 17:57:24.032783 22213 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 17:57:24.038379 22213 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790704375975685-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790704375975685-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40201
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790704375975685-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40201
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790704375975685-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22213
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790704375975685-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:57:24.039502 22213 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:57:24.040534 22213 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:57:24.048991 22221 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:57:24.049120 22213 server_base.cc:1034] running on GCE node
W20260929 17:57:24.049022 22219 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:57:24.048991 22218 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:57:24.049868 22213 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:57:24.050997 22213 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:57:24.052217 22213 hybrid_clock.cc:648] HybridClock initialized: now 1790704644052189 us; error 39 us; skew 500 ppm
Sep 29 17:57:24 dist-test-slave-53nh krb5kdc[22200](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704644, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:57:24.055146 22213 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 17:57:24.056097 22213 webserver.cc:460] Webserver started at http://127.0.98.62:40853/ using document root <none> and password file <none>
I20260929 17:57:24.056646 22213 fs_manager.cc:359] Metadata directory not provided
I20260929 17:57:24.056715 22213 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:57:24.056912 22213 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:57:24.059062 22213 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790704375975685-392-0/minicluster-data/master-0/data/instance:
uuid: "fb779d241e594952af5784968dd522b3"
format_stamp: "Formatted at 2026-09-29 17:57:24 on dist-test-slave-53nh"
server_key: "0556e51839482401eface0d2730d5903"
server_key_iv: "f27287c4cfd53e1345cadde70d0da1ba"
server_key_version: "encryptionkey@0"
I20260929 17:57:24.059623 22213 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790704375975685-392-0/minicluster-data/master-0/wal/instance:
uuid: "fb779d241e594952af5784968dd522b3"
format_stamp: "Formatted at 2026-09-29 17:57:24 on dist-test-slave-53nh"
server_key: "0556e51839482401eface0d2730d5903"
server_key_iv: "f27287c4cfd53e1345cadde70d0da1ba"
server_key_version: "encryptionkey@0"
I20260929 17:57:24.063665 22213 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 17:57:24.066453 22228 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:57:24.067809 22213 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 17:57:24.067924 22213 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790704375975685-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790704375975685-392-0/minicluster-data/master-0/wal
uuid: "fb779d241e594952af5784968dd522b3"
format_stamp: "Formatted at 2026-09-29 17:57:24 on dist-test-slave-53nh"
server_key: "0556e51839482401eface0d2730d5903"
server_key_iv: "f27287c4cfd53e1345cadde70d0da1ba"
server_key_version: "encryptionkey@0"
I20260929 17:57:24.068045 22213 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790704375975685-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790704375975685-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790704375975685-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:57:24.072810 22213 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:57:24.075994 22213 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:57:24.076164 22213 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:57:24.083287 22213 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40201
I20260929 17:57:24.083309 22280 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40201 every 8 connection(s)
I20260929 17:57:24.084241 22213 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790704375975685-392-0/minicluster-data/master-0/data/info.pb
I20260929 17:57:24.087172 22281 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:57:24.092808 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskup3Soe/build/debug/bin/kudu as pid 22213
I20260929 17:57:24.092957 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790704375975685-392-0/minicluster-data/master-0/wal/instance
I20260929 17:57:24.093287 392 external_mini_cluster.cc:1428] Setting key 2f7ccf3213620e2bc586caf859277329
I20260929 17:57:24.093408 22281 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb779d241e594952af5784968dd522b3: Bootstrap starting.
I20260929 17:57:24.095955 22281 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fb779d241e594952af5784968dd522b3: Neither blocks nor log segments found. Creating new log.
I20260929 17:57:24.096728 22281 log.cc:824] T 00000000000000000000000000000000 P fb779d241e594952af5784968dd522b3: Log is configured to *not* fsync() on all Append() calls
I20260929 17:57:24.098956 22284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:24.094799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55659 (local address 127.0.98.62:40201)
0929 17:57:24.095268 (+ 469us) server_negotiation.cc:206] Beginning negotiation
0929 17:57:24.095283 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 17:57:24.095320 (+ 37us) server_negotiation.cc:407] Connection header received
0929 17:57:24.095969 (+ 649us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:57:24.095987 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:57:24.096323 (+ 336us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:57:24.096622 (+ 299us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:57:24.097477 (+ 855us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:57:24.097481 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:57:24.097494 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 17:57:24.097506 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:57:24.097577 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 17:57:24.097604 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:57:24.097725 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0929 17:57:24.097801 (+ 76us) server_negotiation.cc:299] Negotiation successful
0929 17:57:24.097994 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":142,"threads_started":1}
I20260929 17:57:24.099056 22281 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb779d241e594952af5784968dd522b3: No bootstrap required, opened a new log
I20260929 17:57:24.101965 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 17:57:24.103142 22281 raft_consensus.cc:348] T 00000000000000000000000000000000 P fb779d241e594952af5784968dd522b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb779d241e594952af5784968dd522b3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40201 } }
I20260929 17:57:24.103459 22281 raft_consensus.cc:374] T 00000000000000000000000000000000 P fb779d241e594952af5784968dd522b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:57:24.103520 22281 raft_consensus.cc:729] T 00000000000000000000000000000000 P fb779d241e594952af5784968dd522b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb779d241e594952af5784968dd522b3, State: Initialized, Role: FOLLOWER
I20260929 17:57:24.104060 22281 consensus_queue.cc:260] T 00000000000000000000000000000000 P fb779d241e594952af5784968dd522b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb779d241e594952af5784968dd522b3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40201 } }
I20260929 17:57:24.104324 22281 raft_consensus.cc:388] T 00000000000000000000000000000000 P fb779d241e594952af5784968dd522b3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 17:57:24.104411 22281 raft_consensus.cc:482] T 00000000000000000000000000000000 P fb779d241e594952af5784968dd522b3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 17:57:24.104522 22281 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fb779d241e594952af5784968dd522b3 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:57:24.105679 22281 raft_consensus.cc:504] T 00000000000000000000000000000000 P fb779d241e594952af5784968dd522b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb779d241e594952af5784968dd522b3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40201 } }
I20260929 17:57:24.105882 22284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:24.104128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34725 (local address 127.0.98.62:40201)
0929 17:57:24.104269 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0929 17:57:24.104273 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:57:24.104432 (+ 159us) server_negotiation.cc:407] Connection header received
0929 17:57:24.104566 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:57:24.104569 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:57:24.104609 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:57:24.104680 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:57:24.105488 (+ 808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:57:24.105492 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:57:24.105494 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 17:57:24.105506 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:57:24.105535 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 17:57:24.105546 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:57:24.105621 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0929 17:57:24.105706 (+ 85us) server_negotiation.cc:299] Negotiation successful
0929 17:57:24.105754 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260929 17:57:24.106087 22281 leader_election.cc:302] T 00000000000000000000000000000000 P fb779d241e594952af5784968dd522b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fb779d241e594952af5784968dd522b3; no voters:
I20260929 17:57:24.106427 22281 leader_election.cc:288] T 00000000000000000000000000000000 P fb779d241e594952af5784968dd522b3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 17:57:24.106501 22291 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fb779d241e594952af5784968dd522b3 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 17:57:24.106707 22291 raft_consensus.cc:686] T 00000000000000000000000000000000 P fb779d241e594952af5784968dd522b3 [term 1 LEADER]: Becoming Leader. State: Replica: fb779d241e594952af5784968dd522b3, State: Running, Role: LEADER
I20260929 17:57:24.107054 22291 consensus_queue.cc:237] T 00000000000000000000000000000000 P fb779d241e594952af5784968dd522b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb779d241e594952af5784968dd522b3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40201 } }
I20260929 17:57:24.107879 22281 sys_catalog.cc:564] T 00000000000000000000000000000000 P fb779d241e594952af5784968dd522b3 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 17:57:24.108939 22293 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb779d241e594952af5784968dd522b3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fb779d241e594952af5784968dd522b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb779d241e594952af5784968dd522b3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40201 } } }
I20260929 17:57:24.108978 22292 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb779d241e594952af5784968dd522b3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fb779d241e594952af5784968dd522b3. Latest consensus state: current_term: 1 leader_uuid: "fb779d241e594952af5784968dd522b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb779d241e594952af5784968dd522b3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40201 } } }
I20260929 17:57:24.109090 22293 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb779d241e594952af5784968dd522b3 [sys.catalog]: This master's current role is: LEADER
I20260929 17:57:24.109103 22292 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb779d241e594952af5784968dd522b3 [sys.catalog]: This master's current role is: LEADER
I20260929 17:57:24.109452 22300 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 17:57:24.112655 22300 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 17:57:24.119261 22300 catalog_manager.cc:1347] Generated new cluster ID: ff8cf432684e4152a279dbdc14dabc5d
I20260929 17:57:24.119342 22300 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 17:57:24.131732 22300 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 17:57:24.132100 22300 catalog_manager.cc:1504] Loading token signing keys...
I20260929 17:57:24.141111 22300 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fb779d241e594952af5784968dd522b3: Generated new TSK 0
I20260929 17:57:24.141930 22300 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 17:57:24.144701 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskup3Soe/build/debug/bin/kudu with pid 22213
2026-09-29T17:57:24Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1303 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790704375975685-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 17:57:24 dist-test-slave-53nh krb5kdc[22312](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 17:57:24 dist-test-slave-53nh krb5kdc[22312](info): set up 2 sockets
Sep 29 17:57:24 dist-test-slave-53nh krb5kdc[22312](info): commencing operation
krb5kdc: starting...
W20260929 17:57:25.113131 392 mini_kdc.cc:121] Time spent starting KDC: real 0.949s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790704375975685-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790704375975685-392-0/krb5kdc/test-user.keytab.
Sep 29 17:57:25 dist-test-slave-53nh krb5kdc[22312](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704645, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T17:57:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T17:57:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790704375975685-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790704375975685-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 17:57:25.232368 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskup3Soe/build/debug/bin/kudu
/tmp/dist-test-taskup3Soe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790704375975685-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790704375975685-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790704375975685-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790704375975685-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39305
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790704375975685-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39305
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790704375975685-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790704375975685-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790704375975685-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:57:25.343946 22325 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:57:25.344262 22325 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:57:25.344323 22325 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:57:25.348551 22325 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 17:57:25.348629 22325 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:57:25.348656 22325 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:57:25.348677 22325 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 17:57:25.348699 22325 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 17:57:25.354203 22325 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790704375975685-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790704375975685-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39305
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790704375975685-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39305
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790704375975685-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22325
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790704375975685-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:57:25.355327 22325 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:57:25.356379 22325 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:57:25.364728 22331 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:57:25.364728 22333 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:57:25.364732 22330 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:57:25.365097 22325 server_base.cc:1034] running on GCE node
I20260929 17:57:25.365713 22325 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:57:25.366801 22325 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:57:25.367969 22325 hybrid_clock.cc:648] HybridClock initialized: now 1790704645367953 us; error 31 us; skew 500 ppm
Sep 29 17:57:25 dist-test-slave-53nh krb5kdc[22312](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704645, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:57:25.374046 22325 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 17:57:25.374924 22325 webserver.cc:460] Webserver started at http://127.0.98.62:44695/ using document root <none> and password file <none>
I20260929 17:57:25.375461 22325 fs_manager.cc:359] Metadata directory not provided
I20260929 17:57:25.375525 22325 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:57:25.375717 22325 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:57:25.377769 22325 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790704375975685-392-0/minicluster-data/master-0/data/instance:
uuid: "dec9556bdefe4a1cb6188642903f9069"
format_stamp: "Formatted at 2026-09-29 17:57:25 on dist-test-slave-53nh"
server_key: "5a4b87eea7f2ba3d83f66090052c8bd7"
server_key_iv: "62925b822832b3a30db2b39dc61f44d3"
server_key_version: "encryptionkey@0"
I20260929 17:57:25.378297 22325 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790704375975685-392-0/minicluster-data/master-0/wal/instance:
uuid: "dec9556bdefe4a1cb6188642903f9069"
format_stamp: "Formatted at 2026-09-29 17:57:25 on dist-test-slave-53nh"
server_key: "5a4b87eea7f2ba3d83f66090052c8bd7"
server_key_iv: "62925b822832b3a30db2b39dc61f44d3"
server_key_version: "encryptionkey@0"
I20260929 17:57:25.381804 22325 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 17:57:25.384239 22340 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:57:25.385649 22325 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 17:57:25.385766 22325 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790704375975685-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790704375975685-392-0/minicluster-data/master-0/wal
uuid: "dec9556bdefe4a1cb6188642903f9069"
format_stamp: "Formatted at 2026-09-29 17:57:25 on dist-test-slave-53nh"
server_key: "5a4b87eea7f2ba3d83f66090052c8bd7"
server_key_iv: "62925b822832b3a30db2b39dc61f44d3"
server_key_version: "encryptionkey@0"
I20260929 17:57:25.385874 22325 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790704375975685-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790704375975685-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790704375975685-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:57:25.399066 22325 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:57:25.402367 22325 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:57:25.402540 22325 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:57:25.408811 22325 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39305
I20260929 17:57:25.408897 22392 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39305 every 8 connection(s)
I20260929 17:57:25.409782 22325 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790704375975685-392-0/minicluster-data/master-0/data/info.pb
I20260929 17:57:25.412590 22393 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:57:25.418641 22393 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dec9556bdefe4a1cb6188642903f9069: Bootstrap starting.
I20260929 17:57:25.418898 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskup3Soe/build/debug/bin/kudu as pid 22325
I20260929 17:57:25.419050 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790704375975685-392-0/minicluster-data/master-0/wal/instance
I20260929 17:57:25.419353 392 external_mini_cluster.cc:1428] Setting key 7061adc48dd89017a9dc4aba2f06a1fd
I20260929 17:57:25.421545 22393 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dec9556bdefe4a1cb6188642903f9069: Neither blocks nor log segments found. Creating new log.
I20260929 17:57:25.422330 22393 log.cc:824] T 00000000000000000000000000000000 P dec9556bdefe4a1cb6188642903f9069: Log is configured to *not* fsync() on all Append() calls
I20260929 17:57:25.424573 22393 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dec9556bdefe4a1cb6188642903f9069: No bootstrap required, opened a new log
Sep 29 17:57:25 dist-test-slave-53nh krb5kdc[22312](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704645, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 17:57:25.429191 22393 raft_consensus.cc:348] T 00000000000000000000000000000000 P dec9556bdefe4a1cb6188642903f9069 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dec9556bdefe4a1cb6188642903f9069" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39305 } }
I20260929 17:57:25.429450 22393 raft_consensus.cc:374] T 00000000000000000000000000000000 P dec9556bdefe4a1cb6188642903f9069 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:57:25.429512 22393 raft_consensus.cc:729] T 00000000000000000000000000000000 P dec9556bdefe4a1cb6188642903f9069 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dec9556bdefe4a1cb6188642903f9069, State: Initialized, Role: FOLLOWER
I20260929 17:57:25.429986 22393 consensus_queue.cc:260] T 00000000000000000000000000000000 P dec9556bdefe4a1cb6188642903f9069 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dec9556bdefe4a1cb6188642903f9069" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39305 } }
I20260929 17:57:25.430166 22393 raft_consensus.cc:388] T 00000000000000000000000000000000 P dec9556bdefe4a1cb6188642903f9069 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 17:57:25.430245 22393 raft_consensus.cc:482] T 00000000000000000000000000000000 P dec9556bdefe4a1cb6188642903f9069 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 17:57:25.430363 22393 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dec9556bdefe4a1cb6188642903f9069 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:57:25.431325 22393 raft_consensus.cc:504] T 00000000000000000000000000000000 P dec9556bdefe4a1cb6188642903f9069 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dec9556bdefe4a1cb6188642903f9069" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39305 } }
I20260929 17:57:25.431694 22393 leader_election.cc:302] T 00000000000000000000000000000000 P dec9556bdefe4a1cb6188642903f9069 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dec9556bdefe4a1cb6188642903f9069; no voters:
I20260929 17:57:25.432106 22393 leader_election.cc:288] T 00000000000000000000000000000000 P dec9556bdefe4a1cb6188642903f9069 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 17:57:25.432219 22398 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dec9556bdefe4a1cb6188642903f9069 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 17:57:25.432443 22398 raft_consensus.cc:686] T 00000000000000000000000000000000 P dec9556bdefe4a1cb6188642903f9069 [term 1 LEADER]: Becoming Leader. State: Replica: dec9556bdefe4a1cb6188642903f9069, State: Running, Role: LEADER
I20260929 17:57:25.432825 22398 consensus_queue.cc:237] T 00000000000000000000000000000000 P dec9556bdefe4a1cb6188642903f9069 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dec9556bdefe4a1cb6188642903f9069" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39305 } }
I20260929 17:57:25.433648 22393 sys_catalog.cc:564] T 00000000000000000000000000000000 P dec9556bdefe4a1cb6188642903f9069 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 17:57:25.434218 22398 sys_catalog.cc:455] T 00000000000000000000000000000000 P dec9556bdefe4a1cb6188642903f9069 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dec9556bdefe4a1cb6188642903f9069. Latest consensus state: current_term: 1 leader_uuid: "dec9556bdefe4a1cb6188642903f9069" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dec9556bdefe4a1cb6188642903f9069" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39305 } } }
I20260929 17:57:25.434847 22396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:25.420742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47237 (local address 127.0.98.62:39305)
0929 17:57:25.421177 (+ 435us) server_negotiation.cc:206] Beginning negotiation
0929 17:57:25.421193 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 17:57:25.421229 (+ 36us) server_negotiation.cc:407] Connection header received
0929 17:57:25.421951 (+ 722us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:57:25.421971 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:57:25.422352 (+ 381us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:57:25.422687 (+ 335us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:57:25.423539 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:25.424554 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:25.425803 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:25.426323 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:25.426448 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:25.429529 (+ 3081us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:57:25.429536 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:57:25.429554 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:57:25.429587 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:57:25.431384 (+ 1797us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:57:25.431954 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:57:25.431959 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:57:25.431963 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:57:25.432082 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:57:25.432422 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:57:25.432426 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:57:25.432428 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:57:25.433025 (+ 597us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:57:25.433215 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0929 17:57:25.433531 (+ 316us) server_negotiation.cc:299] Negotiation successful
0929 17:57:25.433836 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":130,"threads_started":1}
I20260929 17:57:25.434890 22398 sys_catalog.cc:458] T 00000000000000000000000000000000 P dec9556bdefe4a1cb6188642903f9069 [sys.catalog]: This master's current role is: LEADER
I20260929 17:57:25.434772 22400 sys_catalog.cc:455] T 00000000000000000000000000000000 P dec9556bdefe4a1cb6188642903f9069 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dec9556bdefe4a1cb6188642903f9069" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dec9556bdefe4a1cb6188642903f9069" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39305 } } }
I20260929 17:57:25.437304 22400 sys_catalog.cc:458] T 00000000000000000000000000000000 P dec9556bdefe4a1cb6188642903f9069 [sys.catalog]: This master's current role is: LEADER
I20260929 17:57:25.437834 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260929 17:57:25.438985 22412 catalog_manager.cc:1558] T 00000000000000000000000000000000 P dec9556bdefe4a1cb6188642903f9069: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 17:57:25.439069 22412 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 17:57:25.439154 22414 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 17:57:25.440043 22414 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 17:57:25.445746 22414 catalog_manager.cc:1347] Generated new cluster ID: 4bdf23a4cce54c9eaaa17f60379ce034
I20260929 17:57:25.445813 22414 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 17:57:25.450546 22396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:25.441457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34577 (local address 127.0.98.62:39305)
0929 17:57:25.441566 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0929 17:57:25.441570 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:57:25.441968 (+ 398us) server_negotiation.cc:407] Connection header received
0929 17:57:25.442090 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:57:25.442093 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:57:25.442141 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:57:25.442224 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:57:25.443081 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:25.443884 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:25.445028 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:25.445392 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:25.445500 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:25.447225 (+ 1725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:57:25.447229 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:57:25.447232 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:57:25.447258 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:57:25.448900 (+ 1642us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:57:25.449386 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:57:25.449390 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:57:25.449392 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:57:25.449480 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:57:25.449731 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:57:25.449734 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:57:25.449736 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:57:25.449932 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:57:25.450051 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0929 17:57:25.450259 (+ 208us) server_negotiation.cc:299] Negotiation successful
0929 17:57:25.450373 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260929 17:57:25.471995 22414 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 17:57:25.473395 22414 catalog_manager.cc:1504] Loading token signing keys...
I20260929 17:57:25.484503 22414 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dec9556bdefe4a1cb6188642903f9069: Generated new TSK 0
I20260929 17:57:25.485399 22414 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 17:57:25.492666 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskup3Soe/build/debug/bin/kudu with pid 22325
2026-09-29T17:57:25Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1347 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790704375975685-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 17:57:25 dist-test-slave-53nh krb5kdc[22424](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 17:57:25 dist-test-slave-53nh krb5kdc[22424](info): set up 2 sockets
Sep 29 17:57:25 dist-test-slave-53nh krb5kdc[22424](info): commencing operation
krb5kdc: starting...
W20260929 17:57:26.441051 392 mini_kdc.cc:121] Time spent starting KDC: real 0.929s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790704375975685-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790704375975685-392-0/krb5kdc/test-user.keytab.
Sep 29 17:57:26 dist-test-slave-53nh krb5kdc[22424](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704646, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T17:57:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T17:57:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790704375975685-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790704375975685-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 17:57:26.560623 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskup3Soe/build/debug/bin/kudu
/tmp/dist-test-taskup3Soe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790704375975685-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790704375975685-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790704375975685-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790704375975685-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38479
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790704375975685-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36767
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38479
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790704375975685-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790704375975685-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790704375975685-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:57:26.670960 22437 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:57:26.671247 22437 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:57:26.671289 22437 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:57:26.675482 22437 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 17:57:26.675561 22437 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:57:26.675580 22437 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:57:26.675595 22437 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 17:57:26.675609 22437 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 17:57:26.681211 22437 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36767
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790704375975685-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790704375975685-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38479
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790704375975685-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38479
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790704375975685-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22437
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790704375975685-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:57:26.682325 22437 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:57:26.683391 22437 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:57:26.691910 22442 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:57:26.691970 22443 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:57:26.692060 22437 server_base.cc:1034] running on GCE node
W20260929 17:57:26.691849 22445 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:57:26.692714 22437 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:57:26.693898 22437 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:57:26.695083 22437 hybrid_clock.cc:648] HybridClock initialized: now 1790704646695062 us; error 33 us; skew 500 ppm
Sep 29 17:57:26 dist-test-slave-53nh krb5kdc[22424](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704646, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:57:26.698078 22437 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 17:57:26.699018 22437 webserver.cc:460] Webserver started at http://127.0.98.62:37675/ using document root <none> and password file <none>
I20260929 17:57:26.699558 22437 fs_manager.cc:359] Metadata directory not provided
I20260929 17:57:26.699618 22437 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:57:26.699803 22437 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:57:26.701924 22437 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790704375975685-392-0/minicluster-data/master-0/data/instance:
uuid: "88ce351b84d244bd98b0463c0e4f1681"
format_stamp: "Formatted at 2026-09-29 17:57:26 on dist-test-slave-53nh"
server_key: "822eba589752455926dc96a03068a11b"
server_key_iv: "dd049c2022c8b5e293840db0bd3d5f59"
server_key_version: "encryptionkey@0"
I20260929 17:57:26.702481 22437 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790704375975685-392-0/minicluster-data/master-0/wal/instance:
uuid: "88ce351b84d244bd98b0463c0e4f1681"
format_stamp: "Formatted at 2026-09-29 17:57:26 on dist-test-slave-53nh"
server_key: "822eba589752455926dc96a03068a11b"
server_key_iv: "dd049c2022c8b5e293840db0bd3d5f59"
server_key_version: "encryptionkey@0"
I20260929 17:57:26.706526 22437 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 17:57:26.709225 22452 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:57:26.710659 22437 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 17:57:26.710856 22437 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790704375975685-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790704375975685-392-0/minicluster-data/master-0/wal
uuid: "88ce351b84d244bd98b0463c0e4f1681"
format_stamp: "Formatted at 2026-09-29 17:57:26 on dist-test-slave-53nh"
server_key: "822eba589752455926dc96a03068a11b"
server_key_iv: "dd049c2022c8b5e293840db0bd3d5f59"
server_key_version: "encryptionkey@0"
I20260929 17:57:26.711007 22437 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790704375975685-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790704375975685-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790704375975685-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:57:26.730957 22437 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:57:26.734167 22437 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:57:26.734354 22437 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:57:26.740873 22437 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38479
I20260929 17:57:26.740903 22504 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38479 every 8 connection(s)
I20260929 17:57:26.741852 22437 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790704375975685-392-0/minicluster-data/master-0/data/info.pb
I20260929 17:57:26.744802 22505 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:57:26.747203 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskup3Soe/build/debug/bin/kudu as pid 22437
I20260929 17:57:26.747335 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790704375975685-392-0/minicluster-data/master-0/wal/instance
I20260929 17:57:26.747625 392 external_mini_cluster.cc:1428] Setting key a8049072bd786f730cf6bc8a1a428b31
I20260929 17:57:26.751250 22505 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88ce351b84d244bd98b0463c0e4f1681: Bootstrap starting.
I20260929 17:57:26.753844 22505 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 88ce351b84d244bd98b0463c0e4f1681: Neither blocks nor log segments found. Creating new log.
I20260929 17:57:26.754683 22505 log.cc:824] T 00000000000000000000000000000000 P 88ce351b84d244bd98b0463c0e4f1681: Log is configured to *not* fsync() on all Append() calls
I20260929 17:57:26.756598 22505 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88ce351b84d244bd98b0463c0e4f1681: No bootstrap required, opened a new log
I20260929 17:57:26.756821 22508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:26.749069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44065 (local address 127.0.98.62:38479)
0929 17:57:26.749509 (+ 440us) server_negotiation.cc:206] Beginning negotiation
0929 17:57:26.749525 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 17:57:26.749562 (+ 37us) server_negotiation.cc:407] Connection header received
0929 17:57:26.750199 (+ 637us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:57:26.750218 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:57:26.750598 (+ 380us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:57:26.750889 (+ 291us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:57:26.751323 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:26.752393 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:26.753604 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:26.754153 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:26.754280 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:26.755119 (+ 839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:57:26.755124 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:57:26.755139 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 17:57:26.755535 (+ 396us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:57:26.755592 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 17:57:26.755614 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:57:26.755699 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0929 17:57:26.755839 (+ 140us) server_negotiation.cc:299] Negotiation successful
0929 17:57:26.756050 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":296,"thread_start_us":130,"threads_started":1}
I20260929 17:57:26.759632 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 17:57:26.760461 22505 raft_consensus.cc:348] T 00000000000000000000000000000000 P 88ce351b84d244bd98b0463c0e4f1681 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88ce351b84d244bd98b0463c0e4f1681" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38479 } }
I20260929 17:57:26.760689 22505 raft_consensus.cc:374] T 00000000000000000000000000000000 P 88ce351b84d244bd98b0463c0e4f1681 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:57:26.760772 22505 raft_consensus.cc:729] T 00000000000000000000000000000000 P 88ce351b84d244bd98b0463c0e4f1681 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88ce351b84d244bd98b0463c0e4f1681, State: Initialized, Role: FOLLOWER
I20260929 17:57:26.761240 22505 consensus_queue.cc:260] T 00000000000000000000000000000000 P 88ce351b84d244bd98b0463c0e4f1681 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88ce351b84d244bd98b0463c0e4f1681" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38479 } }
I20260929 17:57:26.761376 22505 raft_consensus.cc:388] T 00000000000000000000000000000000 P 88ce351b84d244bd98b0463c0e4f1681 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 17:57:26.761447 22505 raft_consensus.cc:482] T 00000000000000000000000000000000 P 88ce351b84d244bd98b0463c0e4f1681 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 17:57:26.761543 22505 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 88ce351b84d244bd98b0463c0e4f1681 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:57:26.762696 22505 raft_consensus.cc:504] T 00000000000000000000000000000000 P 88ce351b84d244bd98b0463c0e4f1681 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88ce351b84d244bd98b0463c0e4f1681" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38479 } }
I20260929 17:57:26.763065 22505 leader_election.cc:302] T 00000000000000000000000000000000 P 88ce351b84d244bd98b0463c0e4f1681 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 88ce351b84d244bd98b0463c0e4f1681; no voters:
I20260929 17:57:26.763370 22505 leader_election.cc:288] T 00000000000000000000000000000000 P 88ce351b84d244bd98b0463c0e4f1681 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 17:57:26.763473 22515 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 88ce351b84d244bd98b0463c0e4f1681 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 17:57:26.763667 22515 raft_consensus.cc:686] T 00000000000000000000000000000000 P 88ce351b84d244bd98b0463c0e4f1681 [term 1 LEADER]: Becoming Leader. State: Replica: 88ce351b84d244bd98b0463c0e4f1681, State: Running, Role: LEADER
I20260929 17:57:26.764053 22515 consensus_queue.cc:237] T 00000000000000000000000000000000 P 88ce351b84d244bd98b0463c0e4f1681 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88ce351b84d244bd98b0463c0e4f1681" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38479 } }
I20260929 17:57:26.764672 22505 sys_catalog.cc:564] T 00000000000000000000000000000000 P 88ce351b84d244bd98b0463c0e4f1681 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 17:57:26.765599 22508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:26.761519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38869 (local address 127.0.98.62:38479)
0929 17:57:26.761630 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0929 17:57:26.761634 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:57:26.761759 (+ 125us) server_negotiation.cc:407] Connection header received
0929 17:57:26.761841 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:57:26.761843 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:57:26.761893 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:57:26.761974 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:57:26.762181 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:26.762889 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:26.763869 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:26.764190 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:26.764297 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:26.765144 (+ 847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:57:26.765148 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:57:26.765150 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 17:57:26.765163 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:57:26.765190 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 17:57:26.765203 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:57:26.765245 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0929 17:57:26.765408 (+ 163us) server_negotiation.cc:299] Negotiation successful
0929 17:57:26.765467 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260929 17:57:26.766124 22517 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88ce351b84d244bd98b0463c0e4f1681 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "88ce351b84d244bd98b0463c0e4f1681" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88ce351b84d244bd98b0463c0e4f1681" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38479 } } }
I20260929 17:57:26.766240 22517 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88ce351b84d244bd98b0463c0e4f1681 [sys.catalog]: This master's current role is: LEADER
I20260929 17:57:26.766607 22516 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88ce351b84d244bd98b0463c0e4f1681 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88ce351b84d244bd98b0463c0e4f1681. Latest consensus state: current_term: 1 leader_uuid: "88ce351b84d244bd98b0463c0e4f1681" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88ce351b84d244bd98b0463c0e4f1681" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38479 } } }
I20260929 17:57:26.766920 22516 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88ce351b84d244bd98b0463c0e4f1681 [sys.catalog]: This master's current role is: LEADER
W20260929 17:57:26.769227 22530 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 88ce351b84d244bd98b0463c0e4f1681: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 17:57:26.769289 22530 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 17:57:26.769362 22528 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 17:57:26.770021 22528 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 17:57:26.775390 22528 catalog_manager.cc:1347] Generated new cluster ID: bc6a753124f140b5abb31292f58156a7
I20260929 17:57:26.775465 22528 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 17:57:26.788075 22528 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 17:57:26.789091 22528 catalog_manager.cc:1504] Loading token signing keys...
I20260929 17:57:26.801609 22528 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 88ce351b84d244bd98b0463c0e4f1681: Generated new TSK 0
I20260929 17:57:26.802338 22528 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 17:57:26.808410 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskup3Soe/build/debug/bin/kudu with pid 22437
2026-09-29T17:57:26Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1315 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790704375975685-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 17:57:26 dist-test-slave-53nh krb5kdc[22536](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 17:57:26 dist-test-slave-53nh krb5kdc[22536](info): set up 2 sockets
Sep 29 17:57:26 dist-test-slave-53nh krb5kdc[22536](info): commencing operation
krb5kdc: starting...
W20260929 17:57:27.753309 392 mini_kdc.cc:121] Time spent starting KDC: real 0.926s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790704375975685-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790704375975685-392-0/krb5kdc/test-user.keytab.
Sep 29 17:57:27 dist-test-slave-53nh krb5kdc[22536](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704647, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T17:57:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T17:57:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790704375975685-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790704375975685-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 17:57:27.872182 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskup3Soe/build/debug/bin/kudu
/tmp/dist-test-taskup3Soe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790704375975685-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790704375975685-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790704375975685-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790704375975685-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33241
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790704375975685-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33241
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790704375975685-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790704375975685-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790704375975685-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:57:27.982697 22549 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:57:27.983036 22549 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:57:27.983106 22549 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:57:27.987316 22549 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 17:57:27.987408 22549 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:57:27.987442 22549 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:57:27.987468 22549 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 17:57:27.987494 22549 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 17:57:27.993198 22549 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790704375975685-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790704375975685-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33241
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790704375975685-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33241
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790704375975685-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22549
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790704375975685-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:57:27.994382 22549 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:57:27.995455 22549 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:57:28.004498 22554 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:57:28.004621 22557 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:57:28.004662 22549 server_base.cc:1034] running on GCE node
W20260929 17:57:28.004591 22555 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:57:28.005410 22549 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:57:28.006635 22549 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:57:28.007826 22549 hybrid_clock.cc:648] HybridClock initialized: now 1790704648007803 us; error 36 us; skew 500 ppm
Sep 29 17:57:28 dist-test-slave-53nh krb5kdc[22536](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704648, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:57:28.010645 22549 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 17:57:28.011598 22549 webserver.cc:460] Webserver started at http://127.0.98.62:35211/ using document root <none> and password file <none>
I20260929 17:57:28.012153 22549 fs_manager.cc:359] Metadata directory not provided
I20260929 17:57:28.012223 22549 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:57:28.012424 22549 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:57:28.014588 22549 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790704375975685-392-0/minicluster-data/master-0/data/instance:
uuid: "e92dc03229dd484586750f1398f53512"
format_stamp: "Formatted at 2026-09-29 17:57:28 on dist-test-slave-53nh"
server_key: "5c314e44fa46787a9f8e2fea311dfd0d"
server_key_iv: "af38b36d5af14ef4b9bdf346c92d2710"
server_key_version: "encryptionkey@0"
I20260929 17:57:28.015149 22549 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790704375975685-392-0/minicluster-data/master-0/wal/instance:
uuid: "e92dc03229dd484586750f1398f53512"
format_stamp: "Formatted at 2026-09-29 17:57:28 on dist-test-slave-53nh"
server_key: "5c314e44fa46787a9f8e2fea311dfd0d"
server_key_iv: "af38b36d5af14ef4b9bdf346c92d2710"
server_key_version: "encryptionkey@0"
I20260929 17:57:28.019093 22549 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 17:57:28.021719 22564 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:57:28.022996 22549 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 17:57:28.023118 22549 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790704375975685-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790704375975685-392-0/minicluster-data/master-0/wal
uuid: "e92dc03229dd484586750f1398f53512"
format_stamp: "Formatted at 2026-09-29 17:57:28 on dist-test-slave-53nh"
server_key: "5c314e44fa46787a9f8e2fea311dfd0d"
server_key_iv: "af38b36d5af14ef4b9bdf346c92d2710"
server_key_version: "encryptionkey@0"
I20260929 17:57:28.023223 22549 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790704375975685-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790704375975685-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790704375975685-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:57:28.027989 22549 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:57:28.031141 22549 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:57:28.031322 22549 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:57:28.038429 22549 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33241
I20260929 17:57:28.038460 22616 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33241 every 8 connection(s)
I20260929 17:57:28.039359 22549 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790704375975685-392-0/minicluster-data/master-0/data/info.pb
I20260929 17:57:28.042129 22617 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:57:28.048362 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskup3Soe/build/debug/bin/kudu as pid 22549
I20260929 17:57:28.048503 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790704375975685-392-0/minicluster-data/master-0/wal/instance
I20260929 17:57:28.048549 22617 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e92dc03229dd484586750f1398f53512: Bootstrap starting.
I20260929 17:57:28.048760 392 external_mini_cluster.cc:1428] Setting key 761b646ed06c5250b5a405c01b37d727
I20260929 17:57:28.051105 22617 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e92dc03229dd484586750f1398f53512: Neither blocks nor log segments found. Creating new log.
I20260929 17:57:28.051841 22617 log.cc:824] T 00000000000000000000000000000000 P e92dc03229dd484586750f1398f53512: Log is configured to *not* fsync() on all Append() calls
I20260929 17:57:28.054086 22620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:28.050057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39965 (local address 127.0.98.62:33241)
0929 17:57:28.050505 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0929 17:57:28.050519 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 17:57:28.050554 (+ 35us) server_negotiation.cc:407] Connection header received
0929 17:57:28.051196 (+ 642us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:57:28.051221 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:57:28.051532 (+ 311us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:57:28.051821 (+ 289us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:57:28.052632 (+ 811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:57:28.052637 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:57:28.052653 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 17:57:28.052666 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:57:28.052734 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 17:57:28.052760 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:57:28.052872 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0929 17:57:28.053047 (+ 175us) server_negotiation.cc:299] Negotiation successful
0929 17:57:28.053198 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":292,"thread_start_us":136,"threads_started":1}
I20260929 17:57:28.054167 22617 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e92dc03229dd484586750f1398f53512: No bootstrap required, opened a new log
I20260929 17:57:28.056929 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 17:57:28.058068 22617 raft_consensus.cc:348] T 00000000000000000000000000000000 P e92dc03229dd484586750f1398f53512 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e92dc03229dd484586750f1398f53512" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33241 } }
I20260929 17:57:28.058290 22617 raft_consensus.cc:374] T 00000000000000000000000000000000 P e92dc03229dd484586750f1398f53512 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 17:57:28.058379 22617 raft_consensus.cc:729] T 00000000000000000000000000000000 P e92dc03229dd484586750f1398f53512 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e92dc03229dd484586750f1398f53512, State: Initialized, Role: FOLLOWER
I20260929 17:57:28.058813 22617 consensus_queue.cc:260] T 00000000000000000000000000000000 P e92dc03229dd484586750f1398f53512 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e92dc03229dd484586750f1398f53512" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33241 } }
I20260929 17:57:28.058960 22617 raft_consensus.cc:388] T 00000000000000000000000000000000 P e92dc03229dd484586750f1398f53512 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 17:57:28.059075 22617 raft_consensus.cc:482] T 00000000000000000000000000000000 P e92dc03229dd484586750f1398f53512 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 17:57:28.059178 22617 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e92dc03229dd484586750f1398f53512 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:57:28.060281 22617 raft_consensus.cc:504] T 00000000000000000000000000000000 P e92dc03229dd484586750f1398f53512 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e92dc03229dd484586750f1398f53512" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33241 } }
I20260929 17:57:28.060519 22620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:28.058938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57059 (local address 127.0.98.62:33241)
0929 17:57:28.059047 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0929 17:57:28.059051 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:57:28.059225 (+ 174us) server_negotiation.cc:407] Connection header received
0929 17:57:28.059349 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:57:28.059352 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:57:28.059386 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:57:28.059466 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:57:28.060167 (+ 701us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:57:28.060171 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:57:28.060174 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 17:57:28.060185 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:57:28.060214 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 17:57:28.060226 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:57:28.060322 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0929 17:57:28.060375 (+ 53us) server_negotiation.cc:299] Negotiation successful
0929 17:57:28.060411 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260929 17:57:28.060655 22617 leader_election.cc:302] T 00000000000000000000000000000000 P e92dc03229dd484586750f1398f53512 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e92dc03229dd484586750f1398f53512; no voters:
I20260929 17:57:28.060940 22617 leader_election.cc:288] T 00000000000000000000000000000000 P e92dc03229dd484586750f1398f53512 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 17:57:28.061043 22627 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e92dc03229dd484586750f1398f53512 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 17:57:28.061237 22627 raft_consensus.cc:686] T 00000000000000000000000000000000 P e92dc03229dd484586750f1398f53512 [term 1 LEADER]: Becoming Leader. State: Replica: e92dc03229dd484586750f1398f53512, State: Running, Role: LEADER
I20260929 17:57:28.061590 22627 consensus_queue.cc:237] T 00000000000000000000000000000000 P e92dc03229dd484586750f1398f53512 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e92dc03229dd484586750f1398f53512" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33241 } }
I20260929 17:57:28.062255 22617 sys_catalog.cc:564] T 00000000000000000000000000000000 P e92dc03229dd484586750f1398f53512 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 17:57:28.062819 22627 sys_catalog.cc:455] T 00000000000000000000000000000000 P e92dc03229dd484586750f1398f53512 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e92dc03229dd484586750f1398f53512. Latest consensus state: current_term: 1 leader_uuid: "e92dc03229dd484586750f1398f53512" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e92dc03229dd484586750f1398f53512" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33241 } } }
I20260929 17:57:28.062996 22627 sys_catalog.cc:458] T 00000000000000000000000000000000 P e92dc03229dd484586750f1398f53512 [sys.catalog]: This master's current role is: LEADER
I20260929 17:57:28.063213 22628 sys_catalog.cc:455] T 00000000000000000000000000000000 P e92dc03229dd484586750f1398f53512 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e92dc03229dd484586750f1398f53512" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e92dc03229dd484586750f1398f53512" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33241 } } }
I20260929 17:57:28.063315 22628 sys_catalog.cc:458] T 00000000000000000000000000000000 P e92dc03229dd484586750f1398f53512 [sys.catalog]: This master's current role is: LEADER
I20260929 17:57:28.063416 22634 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 17:57:28.066900 22634 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 17:57:28.072896 22634 catalog_manager.cc:1347] Generated new cluster ID: 4dcf80fe231f4ba48b646a6fbf6e5688
I20260929 17:57:28.072983 22634 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 17:57:28.094587 22634 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 17:57:28.094947 22634 catalog_manager.cc:1504] Loading token signing keys...
I20260929 17:57:28.102330 22634 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e92dc03229dd484586750f1398f53512: Generated new TSK 0
I20260929 17:57:28.103179 22634 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 17:57:28.112413 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskup3Soe/build/debug/bin/kudu with pid 22549
2026-09-29T17:57:28Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1304 ms)
[----------] 18 tests from AuthTokenIssuingTest (23795 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790704375975685-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 17:57:28 dist-test-slave-53nh krb5kdc[22648](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 17:57:28 dist-test-slave-53nh krb5kdc[22648](info): set up 2 sockets
Sep 29 17:57:28 dist-test-slave-53nh krb5kdc[22648](info): commencing operation
krb5kdc: starting...
W20260929 17:57:29.073195 392 mini_kdc.cc:121] Time spent starting KDC: real 0.942s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790704375975685-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790704375975685-392-0/krb5kdc/test-user.keytab.
Sep 29 17:57:29 dist-test-slave-53nh krb5kdc[22648](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704649, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T17:57:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T17:57:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790704375975685-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790704375975685-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 17:57:29.196023 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskup3Soe/build/debug/bin/kudu
/tmp/dist-test-taskup3Soe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790704375975685-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790704375975685-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790704375975685-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790704375975685-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43861
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790704375975685-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32947
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43861,127.0.98.61:35953,127.0.98.60:39169
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790704375975685-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790704375975685-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790704375975685-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:57:29.307708 22661 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 17:57:29.308040 22661 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:57:29.308064 22661 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:57:29.308102 22661 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:57:29.312266 22661 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 17:57:29.312341 22661 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:57:29.312361 22661 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:57:29.312382 22661 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 17:57:29.312397 22661 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 17:57:29.317909 22661 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32947
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790704375975685-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790704375975685-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43861,127.0.98.61:35953,127.0.98.60:39169
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790704375975685-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43861
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790704375975685-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22661
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790704375975685-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:57:29.319049 22661 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:57:29.320143 22661 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:57:29.328732 22669 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:57:29.328831 22661 server_base.cc:1034] running on GCE node
W20260929 17:57:29.328727 22667 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:57:29.328727 22666 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:57:29.329660 22661 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:57:29.330814 22661 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:57:29.331986 22661 hybrid_clock.cc:648] HybridClock initialized: now 1790704649331939 us; error 60 us; skew 500 ppm
Sep 29 17:57:29 dist-test-slave-53nh krb5kdc[22648](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704649, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:57:29.335042 22661 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 17:57:29.336123 22661 webserver.cc:460] Webserver started at http://127.0.98.62:41647/ using document root <none> and password file <none>
I20260929 17:57:29.336771 22661 fs_manager.cc:359] Metadata directory not provided
I20260929 17:57:29.336844 22661 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:57:29.337121 22661 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:57:29.339355 22661 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790704375975685-392-0/minicluster-data/master-0/data/instance:
uuid: "17eeece19fda4a509b8f3bb514630d83"
format_stamp: "Formatted at 2026-09-29 17:57:29 on dist-test-slave-53nh"
server_key: "871748db44a9d815da3e2dbc43485415"
server_key_iv: "43ce8c75e1686c58b2754f50fa31539d"
server_key_version: "encryptionkey@0"
I20260929 17:57:29.339936 22661 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790704375975685-392-0/minicluster-data/master-0/wal/instance:
uuid: "17eeece19fda4a509b8f3bb514630d83"
format_stamp: "Formatted at 2026-09-29 17:57:29 on dist-test-slave-53nh"
server_key: "871748db44a9d815da3e2dbc43485415"
server_key_iv: "43ce8c75e1686c58b2754f50fa31539d"
server_key_version: "encryptionkey@0"
I20260929 17:57:29.343863 22661 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 17:57:29.346624 22676 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:57:29.347985 22661 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 17:57:29.348131 22661 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790704375975685-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790704375975685-392-0/minicluster-data/master-0/wal
uuid: "17eeece19fda4a509b8f3bb514630d83"
format_stamp: "Formatted at 2026-09-29 17:57:29 on dist-test-slave-53nh"
server_key: "871748db44a9d815da3e2dbc43485415"
server_key_iv: "43ce8c75e1686c58b2754f50fa31539d"
server_key_version: "encryptionkey@0"
I20260929 17:57:29.348248 22661 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790704375975685-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790704375975685-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790704375975685-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:57:29.359951 22661 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:57:29.363180 22661 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:57:29.363374 22661 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:57:29.370064 22661 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43861
I20260929 17:57:29.370358 22728 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43861 every 8 connection(s)
I20260929 17:57:29.371026 22661 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790704375975685-392-0/minicluster-data/master-0/data/info.pb
I20260929 17:57:29.372329 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskup3Soe/build/debug/bin/kudu as pid 22661
I20260929 17:57:29.372452 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790704375975685-392-0/minicluster-data/master-0/wal/instance
I20260929 17:57:29.372741 392 external_mini_cluster.cc:1428] Setting key ad3d62f16e83f23ff014079669627e3f
I20260929 17:57:29.374351 22729 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:57:29.379182 22729 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43861 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Sep 29 17:57:29 dist-test-slave-53nh krb5kdc[22648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704649, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 17:57:29.395763 22733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:29.381949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47186 (local address 127.0.98.62:43861)
0929 17:57:29.382484 (+ 535us) server_negotiation.cc:206] Beginning negotiation
0929 17:57:29.382503 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0929 17:57:29.382664 (+ 161us) server_negotiation.cc:407] Connection header received
0929 17:57:29.383731 (+ 1067us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:57:29.383736 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:57:29.383979 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:57:29.384160 (+ 181us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:57:29.385228 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:29.386562 (+ 1334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:29.387660 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:29.388253 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:29.388431 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:29.390871 (+ 2440us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:57:29.390879 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:57:29.390897 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:57:29.390926 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:57:29.392545 (+ 1619us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:57:29.393072 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:57:29.393078 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:57:29.393080 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:57:29.393203 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:57:29.393575 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:57:29.393577 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:57:29.393578 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:57:29.394110 (+ 532us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:57:29.394242 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0929 17:57:29.394639 (+ 397us) server_negotiation.cc:299] Negotiation successful
0929 17:57:29.394917 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":33,"server-negotiator.queue_time_us":335,"spinlock_wait_cycles":76544,"thread_start_us":154,"threads_started":1}
I20260929 17:57:29.395767 22732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:29.381949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43861 (local address 127.0.0.1:47186)
0929 17:57:29.382502 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0929 17:57:29.382553 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0929 17:57:29.383425 (+ 872us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:57:29.384209 (+ 784us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:57:29.384215 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:57:29.384681 (+ 466us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:57:29.385068 (+ 387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:57:29.385089 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:57:29.386705 (+ 1616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:57:29.386708 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:57:29.387516 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:57:29.387526 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:57:29.388442 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:57:29.388445 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:57:29.388541 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:29.389172 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:57:29.389190 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:57:29.390659 (+ 1469us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:57:29.392655 (+ 1996us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:57:29.392663 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:57:29.392673 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:57:29.392921 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:57:29.393326 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:57:29.393330 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:57:29.393332 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:57:29.393484 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:57:29.394246 (+ 762us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:57:29.394252 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:57:29.394388 (+ 136us) client_negotiation.cc:769] Sending connection context
0929 17:57:29.394557 (+ 169us) client_negotiation.cc:240] Negotiation successful
0929 17:57:29.394984 (+ 427us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":345,"mutex_wait_us":114,"thread_start_us":154,"threads_started":1}
I20260929 17:57:29.398217 22729 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35953 } has no permanent_uuid. Determining permanent_uuid...
I20260929 17:57:29.398965 22732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:29.398676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35953 (local address 127.0.0.1:47126)
0929 17:57:29.398827 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0929 17:57:29.398901 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:35953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260929 17:57:29.399164 22680 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:35953: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790704375975685-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790704375975685-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260929 17:57:29.400034 22732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:29.399800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35953 (local address 127.0.0.1:47138)
0929 17:57:29.399916 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0929 17:57:29.399968 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:35953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260929 17:57:29.400187 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskup3Soe/build/debug/bin/kudu
/tmp/dist-test-taskup3Soe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790704375975685-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790704375975685-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790704375975685-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790704375975685-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:35953
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790704375975685-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32947
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43861,127.0.98.61:35953,127.0.98.60:39169
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790704375975685-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790704375975685-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790704375975685-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:57:29.400193 22729 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:35953: Network error: Client connection negotiation failed: client connection to 127.0.98.61:35953: connect: Connection refused (error 111)
I20260929 17:57:29.449463 22729 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35953 } attempt: 1
I20260929 17:57:29.450443 22732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:29.450092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35953 (local address 127.0.0.1:47146)
0929 17:57:29.450265 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0929 17:57:29.450339 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:35953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260929 17:57:29.451188 22732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:29.450959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35953 (local address 127.0.0.1:47154)
0929 17:57:29.451058 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0929 17:57:29.451119 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:35953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260929 17:57:29.451323 22729 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:35953: Network error: Client connection negotiation failed: client connection to 127.0.98.61:35953: connect: Connection refused (error 111)
W20260929 17:57:29.511428 22736 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 17:57:29.511754 22736 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:57:29.511778 22736 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:57:29.511824 22736 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:57:29.515906 22736 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 17:57:29.515967 22736 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:57:29.515986 22736 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:57:29.516001 22736 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 17:57:29.516014 22736 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 17:57:29.519508 22729 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35953 } attempt: 2
I20260929 17:57:29.520395 22732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:29.520081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35953 (local address 127.0.0.1:47156)
0929 17:57:29.520240 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0929 17:57:29.520307 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:35953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260929 17:57:29.521093 22732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:29.520856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35953 (local address 127.0.0.1:47166)
0929 17:57:29.520947 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0929 17:57:29.521021 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:35953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260929 17:57:29.521217 22729 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:35953: Network error: Client connection negotiation failed: client connection to 127.0.98.61:35953: connect: Connection refused (error 111)
I20260929 17:57:29.521837 22736 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32947
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790704375975685-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790704375975685-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43861,127.0.98.61:35953,127.0.98.60:39169
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790704375975685-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:35953
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790704375975685-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22736
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790704375975685-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:57:29.522929 22736 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:57:29.523955 22736 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:57:29.532299 22742 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:57:29.532382 22736 server_base.cc:1034] running on GCE node
W20260929 17:57:29.532299 22741 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:57:29.532330 22744 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:57:29.533142 22736 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:57:29.534276 22736 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:57:29.535467 22736 hybrid_clock.cc:648] HybridClock initialized: now 1790704649535428 us; error 36 us; skew 500 ppm
Sep 29 17:57:29 dist-test-slave-53nh krb5kdc[22648](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704649, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:57:29.538340 22736 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260929 17:57:29.539261 22736 webserver.cc:460] Webserver started at http://127.0.98.61:44399/ using document root <none> and password file <none>
I20260929 17:57:29.539806 22736 fs_manager.cc:359] Metadata directory not provided
I20260929 17:57:29.539873 22736 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:57:29.540067 22736 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:57:29.542192 22736 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790704375975685-392-0/minicluster-data/master-1/data/instance:
uuid: "919eecd744334eb687bad6598efa11ed"
format_stamp: "Formatted at 2026-09-29 17:57:29 on dist-test-slave-53nh"
server_key: "3593543879cd2a885fd174cc5ae55dfc"
server_key_iv: "0d49526b7bc23849a5c8dd24dda4ebc1"
server_key_version: "encryptionkey@0"
I20260929 17:57:29.542753 22736 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790704375975685-392-0/minicluster-data/master-1/wal/instance:
uuid: "919eecd744334eb687bad6598efa11ed"
format_stamp: "Formatted at 2026-09-29 17:57:29 on dist-test-slave-53nh"
server_key: "3593543879cd2a885fd174cc5ae55dfc"
server_key_iv: "0d49526b7bc23849a5c8dd24dda4ebc1"
server_key_version: "encryptionkey@0"
I20260929 17:57:29.546746 22736 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 17:57:29.549723 22751 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:57:29.551187 22736 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 17:57:29.551388 22736 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790704375975685-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790704375975685-392-0/minicluster-data/master-1/wal
uuid: "919eecd744334eb687bad6598efa11ed"
format_stamp: "Formatted at 2026-09-29 17:57:29 on dist-test-slave-53nh"
server_key: "3593543879cd2a885fd174cc5ae55dfc"
server_key_iv: "0d49526b7bc23849a5c8dd24dda4ebc1"
server_key_version: "encryptionkey@0"
I20260929 17:57:29.551517 22736 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790704375975685-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790704375975685-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790704375975685-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:57:29.563014 22736 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:57:29.566519 22736 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:57:29.566704 22736 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:57:29.573534 22736 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:35953
I20260929 17:57:29.573810 22803 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:35953 every 8 connection(s)
I20260929 17:57:29.574560 22736 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790704375975685-392-0/minicluster-data/master-1/data/info.pb
I20260929 17:57:29.576174 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskup3Soe/build/debug/bin/kudu as pid 22736
I20260929 17:57:29.576300 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790704375975685-392-0/minicluster-data/master-1/wal/instance
I20260929 17:57:29.576587 392 external_mini_cluster.cc:1428] Setting key 1fb97e1253e700a275fb5ee670cf77d6
I20260929 17:57:29.577907 22804 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:57:29.582316 22804 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43861 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Sep 29 17:57:29 dist-test-slave-53nh krb5kdc[22648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704649, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 17:57:29.598177 22733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:29.584732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47194 (local address 127.0.98.62:43861)
0929 17:57:29.584887 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0929 17:57:29.584891 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:57:29.585831 (+ 940us) server_negotiation.cc:407] Connection header received
0929 17:57:29.586857 (+ 1026us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:57:29.586860 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:57:29.586923 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:57:29.586999 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:57:29.588593 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:29.589236 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:29.590880 (+ 1644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:29.591187 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:29.591286 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:29.594477 (+ 3191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:57:29.594481 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:57:29.594483 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:57:29.594512 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:57:29.595913 (+ 1401us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:57:29.596430 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:57:29.596433 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:57:29.596435 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:57:29.596516 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:57:29.596890 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:57:29.596894 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:57:29.596895 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:57:29.597144 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:57:29.597227 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0929 17:57:29.597923 (+ 696us) server_negotiation.cc:299] Negotiation successful
0929 17:57:29.598041 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 17:57:29.599030 22807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:29.585154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43861 (local address 127.0.0.1:47194)
0929 17:57:29.585671 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0929 17:57:29.585719 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0929 17:57:29.586628 (+ 909us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:57:29.587210 (+ 582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:57:29.587218 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:57:29.587676 (+ 458us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:57:29.588400 (+ 724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:57:29.588418 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:57:29.589381 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:57:29.589385 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:57:29.590741 (+ 1356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:57:29.590753 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:57:29.591362 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:57:29.591367 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:57:29.591598 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:29.592279 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:57:29.592302 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:57:29.594275 (+ 1973us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:57:29.596032 (+ 1757us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:57:29.596037 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:57:29.596051 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:57:29.596339 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:57:29.596600 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:57:29.596602 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:57:29.596603 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:57:29.596744 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:57:29.597240 (+ 496us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:57:29.597246 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:57:29.597551 (+ 305us) client_negotiation.cc:769] Sending connection context
0929 17:57:29.597810 (+ 259us) client_negotiation.cc:240] Negotiation successful
0929 17:57:29.598105 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":174,"threads_started":1}
I20260929 17:57:29.600522 22804 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35953 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790704375975685-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790704375975685-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260929 17:57:29.604336 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskup3Soe/build/debug/bin/kudu
/tmp/dist-test-taskup3Soe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790704375975685-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790704375975685-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790704375975685-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790704375975685-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:39169
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790704375975685-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32947
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43861,127.0.98.61:35953,127.0.98.60:39169
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790704375975685-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790704375975685-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790704375975685-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 17:57:29 dist-test-slave-53nh krb5kdc[22648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704649, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260929 17:57:29.609941 22807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:29.601085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35953 (local address 127.0.0.1:47176)
0929 17:57:29.601224 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0929 17:57:29.601238 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 17:57:29.601322 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:57:29.601842 (+ 520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:57:29.601844 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:57:29.601981 (+ 137us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:57:29.602073 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:57:29.602079 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:57:29.603090 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:57:29.603093 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:57:29.603790 (+ 697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:57:29.603802 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:57:29.604381 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:57:29.604384 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:57:29.604528 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:29.604985 (+ 457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:57:29.605022 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:57:29.606265 (+ 1243us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:57:29.608374 (+ 2109us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:57:29.608377 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:57:29.608378 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:57:29.608578 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:57:29.608873 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:57:29.608875 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:57:29.608876 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:57:29.608948 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:57:29.609511 (+ 563us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:57:29.609514 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:57:29.609599 (+ 85us) client_negotiation.cc:769] Sending connection context
0929 17:57:29.609689 (+ 90us) client_negotiation.cc:240] Negotiation successful
0929 17:57:29.609804 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64,"mutex_wait_us":31}
I20260929 17:57:29.610164 22809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:29.601164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47176 (local address 127.0.98.61:35953)
0929 17:57:29.601485 (+ 321us) server_negotiation.cc:206] Beginning negotiation
0929 17:57:29.601493 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 17:57:29.601512 (+ 19us) server_negotiation.cc:407] Connection header received
0929 17:57:29.601548 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:57:29.601553 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:57:29.601733 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:57:29.601839 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:57:29.602168 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:29.603008 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:29.603902 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:29.604275 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:29.604372 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:29.606406 (+ 2034us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:57:29.606415 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:57:29.606427 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:57:29.606449 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:57:29.608272 (+ 1823us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:57:29.608684 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:57:29.608691 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:57:29.608694 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:57:29.608787 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:57:29.609042 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:57:29.609045 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:57:29.609047 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:57:29.609387 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:57:29.609537 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0929 17:57:29.609802 (+ 265us) server_negotiation.cc:299] Negotiation successful
0929 17:57:29.609959 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":19,"server-negotiator.queue_time_us":236,"thread_start_us":132,"threads_started":1}
I20260929 17:57:29.612048 22804 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39169 } has no permanent_uuid. Determining permanent_uuid...
I20260929 17:57:29.612358 22729 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35953 } attempt: 3
I20260929 17:57:29.612751 22807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:29.612457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39169 (local address 127.0.0.1:33398)
0929 17:57:29.612600 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0929 17:57:29.612680 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39169: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
W20260929 17:57:29.613161 22754 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39169: connect: Connection refused (error 111)
I20260929 17:57:29.614064 22807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:29.613824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39169 (local address 127.0.0.1:33412)
0929 17:57:29.613952 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0929 17:57:29.614000 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39169: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
W20260929 17:57:29.614313 22804 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:39169: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39169: connect: Connection refused (error 111)
Sep 29 17:57:29 dist-test-slave-53nh krb5kdc[22648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704649, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260929 17:57:29.621726 22809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:29.612994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47184 (local address 127.0.98.61:35953)
0929 17:57:29.613658 (+ 664us) server_negotiation.cc:206] Beginning negotiation
0929 17:57:29.613661 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 17:57:29.613673 (+ 12us) server_negotiation.cc:407] Connection header received
0929 17:57:29.613701 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:57:29.613703 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:57:29.613737 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:57:29.613804 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:57:29.614300 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:29.614878 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:29.616058 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:29.616291 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:29.616374 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:29.618696 (+ 2322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:57:29.618701 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:57:29.618703 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:57:29.618726 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:57:29.620101 (+ 1375us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:57:29.620518 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:57:29.620521 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:57:29.620523 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:57:29.620603 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:57:29.620855 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:57:29.620907 (+ 52us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:57:29.620908 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:57:29.621132 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:57:29.621216 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0929 17:57:29.621469 (+ 253us) server_negotiation.cc:299] Negotiation successful
0929 17:57:29.621577 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":574}
I20260929 17:57:29.621726 22732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:29.612783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35953 (local address 127.0.0.1:47184)
0929 17:57:29.612912 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0929 17:57:29.612925 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 17:57:29.613038 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:57:29.613869 (+ 831us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:57:29.613872 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:57:29.614060 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:57:29.614193 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:57:29.614201 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:57:29.614997 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:57:29.615001 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:57:29.615946 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:57:29.615956 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:57:29.616395 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:57:29.616399 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:57:29.616506 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:29.617084 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:57:29.617100 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:57:29.618554 (+ 1454us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:57:29.620210 (+ 1656us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:57:29.620214 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:57:29.620217 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:57:29.620431 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:57:29.620682 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:57:29.620685 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:57:29.620695 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:57:29.620778 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:57:29.621217 (+ 439us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:57:29.621219 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:57:29.621317 (+ 98us) client_negotiation.cc:769] Sending connection context
0929 17:57:29.621410 (+ 93us) client_negotiation.cc:240] Negotiation successful
0929 17:57:29.621554 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260929 17:57:29.623284 22729 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39169 } has no permanent_uuid. Determining permanent_uuid...
I20260929 17:57:29.624001 22732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:29.623751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39169 (local address 127.0.0.1:33416)
0929 17:57:29.623868 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0929 17:57:29.623925 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39169: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260929 17:57:29.624656 22732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:29.624421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39169 (local address 127.0.0.1:33426)
0929 17:57:29.624560 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0929 17:57:29.624603 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39169: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
W20260929 17:57:29.624788 22729 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:39169: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39169: connect: Connection refused (error 111)
I20260929 17:57:29.655956 22729 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39169 } attempt: 1
I20260929 17:57:29.656808 22732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:29.656529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39169 (local address 127.0.0.1:33438)
0929 17:57:29.656666 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0929 17:57:29.656728 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39169: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260929 17:57:29.657552 22732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:29.657345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39169 (local address 127.0.0.1:33446)
0929 17:57:29.657447 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0929 17:57:29.657495 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39169: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260929 17:57:29.657688 22729 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:39169: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39169: connect: Connection refused (error 111)
I20260929 17:57:29.663585 22804 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39169 } attempt: 1
I20260929 17:57:29.664358 22807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:29.664050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39169 (local address 127.0.0.1:33452)
0929 17:57:29.664225 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0929 17:57:29.664284 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39169: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260929 17:57:29.665091 22807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:29.664825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39169 (local address 127.0.0.1:33460)
0929 17:57:29.664958 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0929 17:57:29.665031 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39169: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
W20260929 17:57:29.665233 22804 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:39169: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39169: connect: Connection refused (error 111)
W20260929 17:57:29.719509 22810 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 17:57:29.719828 22810 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:57:29.719851 22810 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:57:29.719888 22810 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:57:29.724002 22810 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 17:57:29.724068 22810 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:57:29.724093 22810 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:57:29.724110 22810 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 17:57:29.724124 22810 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 17:57:29.729506 22810 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32947
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790704375975685-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790704375975685-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43861,127.0.98.61:35953,127.0.98.60:39169
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790704375975685-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:39169
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790704375975685-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22810
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790704375975685-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:57:29.730634 22810 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:57:29.731644 22810 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 17:57:29.732847 22729 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39169 } attempt: 2
I20260929 17:57:29.733398 22804 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39169 } attempt: 2
I20260929 17:57:29.733776 22732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:29.733429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39169 (local address 127.0.0.1:33462)
0929 17:57:29.733594 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0929 17:57:29.733669 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39169: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260929 17:57:29.734153 22807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:29.733817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39169 (local address 127.0.0.1:33470)
0929 17:57:29.733973 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0929 17:57:29.734079 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39169: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260929 17:57:29.734592 22732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:29.734331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39169 (local address 127.0.0.1:33476)
0929 17:57:29.734472 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0929 17:57:29.734535 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39169: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
W20260929 17:57:29.734754 22729 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:39169: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39169: connect: Connection refused (error 111)
I20260929 17:57:29.734969 22807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:29.734664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39169 (local address 127.0.0.1:33486)
0929 17:57:29.734873 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0929 17:57:29.734916 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39169: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140}
W20260929 17:57:29.735136 22804 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:39169: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39169: connect: Connection refused (error 111)
W20260929 17:57:29.740801 22816 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:57:29.740801 22819 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:57:29.740803 22817 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:57:29.741029 22810 server_base.cc:1034] running on GCE node
I20260929 17:57:29.741629 22810 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:57:29.742787 22810 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:57:29.743973 22810 hybrid_clock.cc:648] HybridClock initialized: now 1790704649743947 us; error 39 us; skew 500 ppm
Sep 29 17:57:29 dist-test-slave-53nh krb5kdc[22648](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704649, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:57:29.747051 22810 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260929 17:57:29.748131 22810 webserver.cc:460] Webserver started at http://127.0.98.60:40639/ using document root <none> and password file <none>
I20260929 17:57:29.748713 22810 fs_manager.cc:359] Metadata directory not provided
I20260929 17:57:29.748777 22810 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:57:29.749037 22810 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:57:29.751407 22810 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790704375975685-392-0/minicluster-data/master-2/data/instance:
uuid: "895df0fd36404c8a95cd03fb51ae877a"
format_stamp: "Formatted at 2026-09-29 17:57:29 on dist-test-slave-53nh"
server_key: "764c303237bf9f41fae4162030c4f837"
server_key_iv: "7b9fbec190f32252e1b776196799f48a"
server_key_version: "encryptionkey@0"
I20260929 17:57:29.752136 22810 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790704375975685-392-0/minicluster-data/master-2/wal/instance:
uuid: "895df0fd36404c8a95cd03fb51ae877a"
format_stamp: "Formatted at 2026-09-29 17:57:29 on dist-test-slave-53nh"
server_key: "764c303237bf9f41fae4162030c4f837"
server_key_iv: "7b9fbec190f32252e1b776196799f48a"
server_key_version: "encryptionkey@0"
I20260929 17:57:29.756253 22810 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 17:57:29.758961 22826 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:57:29.760282 22810 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 17:57:29.760404 22810 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790704375975685-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790704375975685-392-0/minicluster-data/master-2/wal
uuid: "895df0fd36404c8a95cd03fb51ae877a"
format_stamp: "Formatted at 2026-09-29 17:57:29 on dist-test-slave-53nh"
server_key: "764c303237bf9f41fae4162030c4f837"
server_key_iv: "7b9fbec190f32252e1b776196799f48a"
server_key_version: "encryptionkey@0"
I20260929 17:57:29.760524 22810 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790704375975685-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790704375975685-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790704375975685-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:57:29.780316 22810 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:57:29.783634 22810 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:57:29.783826 22810 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:57:29.791280 22810 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:39169
I20260929 17:57:29.791579 22878 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:39169 every 8 connection(s)
I20260929 17:57:29.792312 22810 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790704375975685-392-0/minicluster-data/master-2/data/info.pb
I20260929 17:57:29.795430 22879 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:57:29.799369 22879 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43861 } has no permanent_uuid. Determining permanent_uuid...
I20260929 17:57:29.801458 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskup3Soe/build/debug/bin/kudu as pid 22810
I20260929 17:57:29.801576 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790704375975685-392-0/minicluster-data/master-2/wal/instance
I20260929 17:57:29.801828 392 external_mini_cluster.cc:1428] Setting key 5c661a181d95b56bd0ce3c0a1aeed21d
Sep 29 17:57:29 dist-test-slave-53nh krb5kdc[22648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704649, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Sep 29 17:57:29 dist-test-slave-53nh krb5kdc[22648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704649, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260929 17:57:29.814285 22733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:29.801252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47208 (local address 127.0.98.62:43861)
0929 17:57:29.801390 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0929 17:57:29.801394 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:57:29.802527 (+ 1133us) server_negotiation.cc:407] Connection header received
0929 17:57:29.803648 (+ 1121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:57:29.803652 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:57:29.803702 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:57:29.803785 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:57:29.804850 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:29.805710 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:29.807283 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:29.807634 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:29.807740 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:29.810536 (+ 2796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:57:29.810540 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:57:29.810543 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:57:29.810573 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:57:29.812089 (+ 1516us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:57:29.812728 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:57:29.812731 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:57:29.812732 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:57:29.812810 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:57:29.813191 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:57:29.813195 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:57:29.813196 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:57:29.813393 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:57:29.813490 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0929 17:57:29.814002 (+ 512us) server_negotiation.cc:299] Negotiation successful
0929 17:57:29.814123 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 17:57:29.815194 22881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:29.801768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43861 (local address 127.0.0.1:47208)
0929 17:57:29.802361 (+ 593us) negotiation.cc:107] Waiting for socket to connect
0929 17:57:29.802425 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0929 17:57:29.803441 (+ 1016us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:57:29.803843 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:57:29.803849 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:57:29.804344 (+ 495us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:57:29.804699 (+ 355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:57:29.804721 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:57:29.805832 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:57:29.805842 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:57:29.807159 (+ 1317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:57:29.807169 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:57:29.807785 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:57:29.807788 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:57:29.807938 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:29.808693 (+ 755us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:57:29.808712 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:57:29.810379 (+ 1667us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:57:29.812208 (+ 1829us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:57:29.812215 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:57:29.812226 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:57:29.812614 (+ 388us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:57:29.812900 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:57:29.812903 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:57:29.812905 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:57:29.813104 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:57:29.813508 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:57:29.813515 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:57:29.813772 (+ 257us) client_negotiation.cc:769] Sending connection context
0929 17:57:29.813923 (+ 151us) client_negotiation.cc:240] Negotiation successful
0929 17:57:29.814242 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":167,"threads_started":1}
I20260929 17:57:29.815197 22882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:29.802503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33488 (local address 127.0.98.60:39169)
0929 17:57:29.802813 (+ 310us) server_negotiation.cc:206] Beginning negotiation
0929 17:57:29.802826 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 17:57:29.803032 (+ 206us) server_negotiation.cc:407] Connection header received
0929 17:57:29.803447 (+ 415us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:57:29.803489 (+ 42us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:57:29.803718 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:57:29.803894 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:57:29.804725 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:29.805967 (+ 1242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:29.807251 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:29.807839 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:29.807933 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:29.810973 (+ 3040us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:57:29.810979 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:57:29.810989 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:57:29.811010 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:57:29.812711 (+ 1701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:57:29.813118 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:57:29.813124 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:57:29.813126 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:57:29.813200 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:57:29.813438 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:57:29.813442 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:57:29.813444 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:57:29.813764 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:57:29.813961 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0929 17:57:29.814159 (+ 198us) server_negotiation.cc:299] Negotiation successful
0929 17:57:29.814294 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":222,"spinlock_wait_cycles":3968,"thread_start_us":96,"threads_started":1}
I20260929 17:57:29.816392 22879 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35953 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 17:57:29 dist-test-slave-53nh krb5kdc[22648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704649, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Sep 29 17:57:29 dist-test-slave-53nh krb5kdc[22648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704649, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 17:57:29.825827 22881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:29.816866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35953 (local address 127.0.0.1:47198)
0929 17:57:29.817185 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0929 17:57:29.817200 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 17:57:29.817281 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:57:29.817579 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:57:29.817581 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:57:29.817762 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:57:29.817888 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:57:29.817895 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:57:29.818877 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:57:29.818880 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:57:29.819596 (+ 716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:57:29.819605 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:57:29.820161 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:57:29.820165 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:57:29.820278 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:29.820881 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:57:29.820893 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:57:29.822423 (+ 1530us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:57:29.824170 (+ 1747us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:57:29.824174 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:57:29.824177 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:57:29.824402 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:57:29.824681 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:57:29.824685 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:57:29.824688 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:57:29.824779 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:57:29.825286 (+ 507us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:57:29.825289 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:57:29.825423 (+ 134us) client_negotiation.cc:769] Sending connection context
0929 17:57:29.825527 (+ 104us) client_negotiation.cc:240] Negotiation successful
0929 17:57:29.825657 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258}
I20260929 17:57:29.825870 22809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:29.817071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47198 (local address 127.0.98.61:35953)
0929 17:57:29.817187 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0929 17:57:29.817191 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:57:29.817240 (+ 49us) server_negotiation.cc:407] Connection header received
0929 17:57:29.817388 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:57:29.817391 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:57:29.817470 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:57:29.817556 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:57:29.818020 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:29.818773 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:29.819702 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:29.820034 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:29.820134 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:29.822573 (+ 2439us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:57:29.822576 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:57:29.822585 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:57:29.822610 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:57:29.824048 (+ 1438us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:57:29.824501 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:57:29.824506 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:57:29.824507 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:57:29.824590 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:57:29.824870 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:57:29.824873 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:57:29.824875 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:57:29.825169 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:57:29.825286 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0929 17:57:29.825599 (+ 313us) server_negotiation.cc:299] Negotiation successful
0929 17:57:29.825738 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260929 17:57:29.826268 22804 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39169 } attempt: 3
I20260929 17:57:29.826726 22879 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39169 } has no permanent_uuid. Determining permanent_uuid...
I20260929 17:57:29.826921 22733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:29.817402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47212 (local address 127.0.98.62:43861)
0929 17:57:29.817513 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0929 17:57:29.817516 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 17:57:29.817605 (+ 89us) server_negotiation.cc:407] Connection header received
0929 17:57:29.817710 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:57:29.817713 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:57:29.817756 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:57:29.817830 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:57:29.818607 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:29.819228 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:29.820144 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:29.820484 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:29.820567 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:29.823236 (+ 2669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:57:29.823239 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:57:29.823241 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:57:29.823266 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:57:29.824587 (+ 1321us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:57:29.825298 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:57:29.825301 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:57:29.825303 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:57:29.825403 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:57:29.825714 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:57:29.825717 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:57:29.825719 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:57:29.825984 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:57:29.826230 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0929 17:57:29.826598 (+ 368us) server_negotiation.cc:299] Negotiation successful
0929 17:57:29.826732 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
Sep 29 17:57:29 dist-test-slave-53nh krb5kdc[22648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704649, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 29 17:57:29 dist-test-slave-53nh krb5kdc[22648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704649, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 29 17:57:29 dist-test-slave-53nh krb5kdc[22648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704649, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260929 17:57:29.833887 22729 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39169 } attempt: 3
I20260929 17:57:29.837352 22807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:29.826681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39169 (local address 127.0.0.1:33490)
0929 17:57:29.826950 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0929 17:57:29.826966 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 17:57:29.827049 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:57:29.827646 (+ 597us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:57:29.827649 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:57:29.827814 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:57:29.827930 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:57:29.827938 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:57:29.828946 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:57:29.828950 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:57:29.829833 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:57:29.829844 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:57:29.830440 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:57:29.830443 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:57:29.830531 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:29.831054 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:57:29.831070 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:57:29.832637 (+ 1567us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:57:29.834524 (+ 1887us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:57:29.834528 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:57:29.834530 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:57:29.835120 (+ 590us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:57:29.836196 (+ 1076us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:57:29.836199 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:57:29.836201 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:57:29.836295 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:57:29.836862 (+ 567us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:57:29.836865 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:57:29.836967 (+ 102us) client_negotiation.cc:769] Sending connection context
0929 17:57:29.837068 (+ 101us) client_negotiation.cc:240] Negotiation successful
0929 17:57:29.837187 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191}
I20260929 17:57:29.837962 22882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:29.827197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33490 (local address 127.0.98.60:39169)
0929 17:57:29.827319 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0929 17:57:29.827323 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:57:29.827352 (+ 29us) server_negotiation.cc:407] Connection header received
0929 17:57:29.827390 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:57:29.827393 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:57:29.827439 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:57:29.827570 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:57:29.828043 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:29.828797 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:29.829967 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:29.830318 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:29.830756 (+ 438us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:29.832764 (+ 2008us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:57:29.832769 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:57:29.832773 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:57:29.832801 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:57:29.834388 (+ 1587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:57:29.835424 (+ 1036us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:57:29.835428 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:57:29.835430 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:57:29.835523 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:57:29.836398 (+ 875us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:57:29.836402 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:57:29.836404 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:57:29.836756 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:57:29.837562 (+ 806us) server_negotiation.cc:1035] Waiting for connection context
0929 17:57:29.837650 (+ 88us) server_negotiation.cc:299] Negotiation successful
0929 17:57:29.837810 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 17:57:29.838487 22809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:29.828366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47214 (local address 127.0.98.61:35953)
0929 17:57:29.828473 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0929 17:57:29.828477 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:57:29.828490 (+ 13us) server_negotiation.cc:407] Connection header received
0929 17:57:29.828578 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:57:29.828582 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:57:29.828621 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:57:29.828700 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:57:29.829484 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:29.829969 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:29.830808 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:29.831025 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:29.831122 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:29.834318 (+ 3196us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:57:29.834322 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:57:29.834325 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:57:29.834358 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:57:29.836092 (+ 1734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:57:29.836541 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:57:29.836544 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:57:29.836546 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:57:29.836616 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:57:29.836901 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:57:29.836905 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:57:29.836907 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:57:29.837172 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:57:29.837592 (+ 420us) server_negotiation.cc:1035] Waiting for connection context
0929 17:57:29.838261 (+ 669us) server_negotiation.cc:299] Negotiation successful
0929 17:57:29.838359 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 17:57:29.839828 22881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:29.827206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39169 (local address 127.0.0.1:33494)
0929 17:57:29.827398 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0929 17:57:29.827411 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 17:57:29.827477 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:57:29.828075 (+ 598us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:57:29.828078 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:57:29.828235 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:57:29.828315 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:57:29.828322 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:57:29.829892 (+ 1570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:57:29.829895 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:57:29.830629 (+ 734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:57:29.830639 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:57:29.831592 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:57:29.831595 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:57:29.831669 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:29.832089 (+ 420us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:57:29.832103 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:57:29.834079 (+ 1976us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:57:29.836552 (+ 2473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:57:29.836555 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:57:29.836557 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:57:29.837903 (+ 1346us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:57:29.838200 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:57:29.838204 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:57:29.838206 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:57:29.838290 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:57:29.838702 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:57:29.838705 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:57:29.839403 (+ 698us) client_negotiation.cc:769] Sending connection context
0929 17:57:29.839515 (+ 112us) client_negotiation.cc:240] Negotiation successful
0929 17:57:29.839652 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"mutex_wait_us":3}
Sep 29 17:57:29 dist-test-slave-53nh krb5kdc[22648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704649, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260929 17:57:29.840309 22884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:29.827370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33494 (local address 127.0.98.60:39169)
0929 17:57:29.827874 (+ 504us) server_negotiation.cc:206] Beginning negotiation
0929 17:57:29.827883 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 17:57:29.827898 (+ 15us) server_negotiation.cc:407] Connection header received
0929 17:57:29.827934 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:57:29.827936 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:57:29.827980 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:57:29.828069 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:57:29.828434 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:29.829781 (+ 1347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:29.831074 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:29.831408 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:29.831497 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:29.834237 (+ 2740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:57:29.834241 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:57:29.834244 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:57:29.834269 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:57:29.836432 (+ 2163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:57:29.838012 (+ 1580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:57:29.838015 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:57:29.838017 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:57:29.838103 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:57:29.838383 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:57:29.838386 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:57:29.838388 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:57:29.838587 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:57:29.838670 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0929 17:57:29.840011 (+ 1341us) server_negotiation.cc:299] Negotiation successful
0929 17:57:29.840130 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":435,"thread_start_us":118,"threads_started":1}
I20260929 17:57:29.841869 22804 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 919eecd744334eb687bad6598efa11ed: Bootstrap starting.
I20260929 17:57:29.842548 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 17:57:29.843621 22879 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 895df0fd36404c8a95cd03fb51ae877a: Bootstrap starting.
I20260929 17:57:29.844746 22804 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 919eecd744334eb687bad6598efa11ed: Neither blocks nor log segments found. Creating new log.
I20260929 17:57:29.845244 22732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:29.834407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39169 (local address 127.0.0.1:33504)
0929 17:57:29.834714 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0929 17:57:29.834726 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 17:57:29.834816 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:57:29.835351 (+ 535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:57:29.835355 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:57:29.835529 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:57:29.835643 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:57:29.835651 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:57:29.836759 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:57:29.836764 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:57:29.837555 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:57:29.837569 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:57:29.838597 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:57:29.838602 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:57:29.838700 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:29.839182 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:57:29.839196 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:57:29.840671 (+ 1475us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:57:29.843470 (+ 2799us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:57:29.843474 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:57:29.843476 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:57:29.843744 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:57:29.844101 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:57:29.844105 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:57:29.844107 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:57:29.844205 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:57:29.844684 (+ 479us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:57:29.844688 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:57:29.844796 (+ 108us) client_negotiation.cc:769] Sending connection context
0929 17:57:29.844886 (+ 90us) client_negotiation.cc:240] Negotiation successful
0929 17:57:29.845067 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":225}
I20260929 17:57:29.843955 22703 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "17eeece19fda4a509b8f3bb514630d83"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47212
I20260929 17:57:29.845842 22885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:29.834695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33504 (local address 127.0.98.60:39169)
0929 17:57:29.835127 (+ 432us) server_negotiation.cc:206] Beginning negotiation
0929 17:57:29.835133 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 17:57:29.835144 (+ 11us) server_negotiation.cc:407] Connection header received
0929 17:57:29.835172 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:57:29.835174 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:57:29.835205 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:57:29.835286 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:57:29.835768 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:29.836588 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:29.837671 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:29.838458 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:29.839081 (+ 623us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:29.840819 (+ 1738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:57:29.840824 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:57:29.840827 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:57:29.840853 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:57:29.843308 (+ 2455us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:57:29.843875 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:57:29.843878 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:57:29.843880 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:57:29.843984 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:57:29.844321 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:57:29.844324 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:57:29.844326 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:57:29.844564 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:57:29.845445 (+ 881us) server_negotiation.cc:1035] Waiting for connection context
0929 17:57:29.845530 (+ 85us) server_negotiation.cc:299] Negotiation successful
0929 17:57:29.845656 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":116,"threads_started":1}
I20260929 17:57:29.845836 22804 log.cc:824] T 00000000000000000000000000000000 P 919eecd744334eb687bad6598efa11ed: Log is configured to *not* fsync() on all Append() calls
I20260929 17:57:29.846896 22879 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 895df0fd36404c8a95cd03fb51ae877a: Neither blocks nor log segments found. Creating new log.
I20260929 17:57:29.847738 22879 log.cc:824] T 00000000000000000000000000000000 P 895df0fd36404c8a95cd03fb51ae877a: Log is configured to *not* fsync() on all Append() calls
I20260929 17:57:29.848677 22729 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 17eeece19fda4a509b8f3bb514630d83: Bootstrap starting.
I20260929 17:57:29.850000 22879 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 895df0fd36404c8a95cd03fb51ae877a: No bootstrap required, opened a new log
I20260929 17:57:29.851262 22729 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 17eeece19fda4a509b8f3bb514630d83: Neither blocks nor log segments found. Creating new log.
I20260929 17:57:29.852123 22729 log.cc:824] T 00000000000000000000000000000000 P 17eeece19fda4a509b8f3bb514630d83: Log is configured to *not* fsync() on all Append() calls
I20260929 17:57:29.853994 22804 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 919eecd744334eb687bad6598efa11ed: No bootstrap required, opened a new log
I20260929 17:57:29.854189 22729 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 17eeece19fda4a509b8f3bb514630d83: No bootstrap required, opened a new log
I20260929 17:57:29.854609 22879 raft_consensus.cc:348] T 00000000000000000000000000000000 P 895df0fd36404c8a95cd03fb51ae877a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17eeece19fda4a509b8f3bb514630d83" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43861 } } peers { permanent_uuid: "919eecd744334eb687bad6598efa11ed" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35953 } } peers { permanent_uuid: "895df0fd36404c8a95cd03fb51ae877a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39169 } }
I20260929 17:57:29.854854 22879 raft_consensus.cc:729] T 00000000000000000000000000000000 P 895df0fd36404c8a95cd03fb51ae877a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 895df0fd36404c8a95cd03fb51ae877a, State: Initialized, Role: FOLLOWER
I20260929 17:57:29.855211 22879 consensus_queue.cc:260] T 00000000000000000000000000000000 P 895df0fd36404c8a95cd03fb51ae877a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17eeece19fda4a509b8f3bb514630d83" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43861 } } peers { permanent_uuid: "919eecd744334eb687bad6598efa11ed" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35953 } } peers { permanent_uuid: "895df0fd36404c8a95cd03fb51ae877a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39169 } }
I20260929 17:57:29.855854 22889 sys_catalog.cc:455] T 00000000000000000000000000000000 P 895df0fd36404c8a95cd03fb51ae877a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17eeece19fda4a509b8f3bb514630d83" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43861 } } peers { permanent_uuid: "919eecd744334eb687bad6598efa11ed" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35953 } } peers { permanent_uuid: "895df0fd36404c8a95cd03fb51ae877a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39169 } } }
I20260929 17:57:29.856159 22889 sys_catalog.cc:458] T 00000000000000000000000000000000 P 895df0fd36404c8a95cd03fb51ae877a [sys.catalog]: This master's current role is: FOLLOWER
I20260929 17:57:29.856575 22879 sys_catalog.cc:564] T 00000000000000000000000000000000 P 895df0fd36404c8a95cd03fb51ae877a [sys.catalog]: configured and running, proceeding with master startup.
I20260929 17:57:29.857087 22729 raft_consensus.cc:348] T 00000000000000000000000000000000 P 17eeece19fda4a509b8f3bb514630d83 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17eeece19fda4a509b8f3bb514630d83" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43861 } } peers { permanent_uuid: "919eecd744334eb687bad6598efa11ed" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35953 } } peers { permanent_uuid: "895df0fd36404c8a95cd03fb51ae877a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39169 } }
I20260929 17:57:29.857300 22729 raft_consensus.cc:729] T 00000000000000000000000000000000 P 17eeece19fda4a509b8f3bb514630d83 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17eeece19fda4a509b8f3bb514630d83, State: Initialized, Role: FOLLOWER
I20260929 17:57:29.857617 22729 consensus_queue.cc:260] T 00000000000000000000000000000000 P 17eeece19fda4a509b8f3bb514630d83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17eeece19fda4a509b8f3bb514630d83" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43861 } } peers { permanent_uuid: "919eecd744334eb687bad6598efa11ed" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35953 } } peers { permanent_uuid: "895df0fd36404c8a95cd03fb51ae877a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39169 } }
I20260929 17:57:29.857939 22804 raft_consensus.cc:348] T 00000000000000000000000000000000 P 919eecd744334eb687bad6598efa11ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17eeece19fda4a509b8f3bb514630d83" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43861 } } peers { permanent_uuid: "919eecd744334eb687bad6598efa11ed" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35953 } } peers { permanent_uuid: "895df0fd36404c8a95cd03fb51ae877a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39169 } }
I20260929 17:57:29.858147 22804 raft_consensus.cc:729] T 00000000000000000000000000000000 P 919eecd744334eb687bad6598efa11ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 919eecd744334eb687bad6598efa11ed, State: Initialized, Role: FOLLOWER
I20260929 17:57:29.858107 22895 sys_catalog.cc:455] T 00000000000000000000000000000000 P 17eeece19fda4a509b8f3bb514630d83 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17eeece19fda4a509b8f3bb514630d83" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43861 } } peers { permanent_uuid: "919eecd744334eb687bad6598efa11ed" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35953 } } peers { permanent_uuid: "895df0fd36404c8a95cd03fb51ae877a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39169 } } }
I20260929 17:57:29.858444 22895 sys_catalog.cc:458] T 00000000000000000000000000000000 P 17eeece19fda4a509b8f3bb514630d83 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 17:57:29.858446 22804 consensus_queue.cc:260] T 00000000000000000000000000000000 P 919eecd744334eb687bad6598efa11ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17eeece19fda4a509b8f3bb514630d83" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43861 } } peers { permanent_uuid: "919eecd744334eb687bad6598efa11ed" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35953 } } peers { permanent_uuid: "895df0fd36404c8a95cd03fb51ae877a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39169 } }
I20260929 17:57:29.858898 22729 sys_catalog.cc:564] T 00000000000000000000000000000000 P 17eeece19fda4a509b8f3bb514630d83 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 17:57:29.858999 22703 raft_consensus.cc:482] T 00000000000000000000000000000000 P 17eeece19fda4a509b8f3bb514630d83 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260929 17:57:29.859097 22703 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 17eeece19fda4a509b8f3bb514630d83 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:57:29.859650 22804 sys_catalog.cc:564] T 00000000000000000000000000000000 P 919eecd744334eb687bad6598efa11ed [sys.catalog]: configured and running, proceeding with master startup.
I20260929 17:57:29.860351 22703 raft_consensus.cc:504] T 00000000000000000000000000000000 P 17eeece19fda4a509b8f3bb514630d83 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17eeece19fda4a509b8f3bb514630d83" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43861 } } peers { permanent_uuid: "919eecd744334eb687bad6598efa11ed" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35953 } } peers { permanent_uuid: "895df0fd36404c8a95cd03fb51ae877a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39169 } }
I20260929 17:57:29.861315 22703 leader_election.cc:288] T 00000000000000000000000000000000 P 17eeece19fda4a509b8f3bb514630d83 [CANDIDATE]: Term 1 election: Requested vote from peers 919eecd744334eb687bad6598efa11ed (127.0.98.61:35953), 895df0fd36404c8a95cd03fb51ae877a (127.0.98.60:39169)
I20260929 17:57:29.862003 22778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "17eeece19fda4a509b8f3bb514630d83" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "919eecd744334eb687bad6598efa11ed"
I20260929 17:57:29.862231 22778 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 919eecd744334eb687bad6598efa11ed [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:57:29.863590 22778 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 919eecd744334eb687bad6598efa11ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17eeece19fda4a509b8f3bb514630d83 in term 1.
I20260929 17:57:29.864013 22680 leader_election.cc:302] T 00000000000000000000000000000000 P 17eeece19fda4a509b8f3bb514630d83 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17eeece19fda4a509b8f3bb514630d83, 919eecd744334eb687bad6598efa11ed; no voters:
W20260929 17:57:29.864951 22682 tablet.cc:2369] T 00000000000000000000000000000000 P 17eeece19fda4a509b8f3bb514630d83: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 17:57:29.868332 22925 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 919eecd744334eb687bad6598efa11ed: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 17:57:29.868443 22925 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 17:57:29.868919 22736 master_runner.cc:232] Existing masters have differing terms: 1 vs 0
I20260929 17:57:29.869069 22736 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 17:57:29.869489 22895 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 17eeece19fda4a509b8f3bb514630d83 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 17:57:29.869462 22902 sys_catalog.cc:455] T 00000000000000000000000000000000 P 919eecd744334eb687bad6598efa11ed [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17eeece19fda4a509b8f3bb514630d83" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43861 } } peers { permanent_uuid: "919eecd744334eb687bad6598efa11ed" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35953 } } peers { permanent_uuid: "895df0fd36404c8a95cd03fb51ae877a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39169 } } }
I20260929 17:57:29.869580 22902 sys_catalog.cc:458] T 00000000000000000000000000000000 P 919eecd744334eb687bad6598efa11ed [sys.catalog]: This master's current role is: FOLLOWER
I20260929 17:57:29.869656 22895 raft_consensus.cc:686] T 00000000000000000000000000000000 P 17eeece19fda4a509b8f3bb514630d83 [term 1 LEADER]: Becoming Leader. State: Replica: 17eeece19fda4a509b8f3bb514630d83, State: Running, Role: LEADER
I20260929 17:57:29.870523 22895 consensus_queue.cc:237] T 00000000000000000000000000000000 P 17eeece19fda4a509b8f3bb514630d83 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17eeece19fda4a509b8f3bb514630d83" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43861 } } peers { permanent_uuid: "919eecd744334eb687bad6598efa11ed" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35953 } } peers { permanent_uuid: "895df0fd36404c8a95cd03fb51ae877a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39169 } }
I20260929 17:57:29.871027 22810 master_runner.cc:263] No leader master found from master 895df0fd36404c8a95cd03fb51ae877a
I20260929 17:57:29.871094 22810 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 17:57:29.871189 22661 master_runner.cc:232] Existing masters have differing terms: 1 vs 0
I20260929 17:57:29.871294 22661 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 17:57:29.872617 22895 sys_catalog.cc:455] T 00000000000000000000000000000000 P 17eeece19fda4a509b8f3bb514630d83 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 17eeece19fda4a509b8f3bb514630d83. Latest consensus state: current_term: 1 leader_uuid: "17eeece19fda4a509b8f3bb514630d83" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17eeece19fda4a509b8f3bb514630d83" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43861 } } peers { permanent_uuid: "919eecd744334eb687bad6598efa11ed" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35953 } } peers { permanent_uuid: "895df0fd36404c8a95cd03fb51ae877a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39169 } } }
I20260929 17:57:29.872735 22895 sys_catalog.cc:458] T 00000000000000000000000000000000 P 17eeece19fda4a509b8f3bb514630d83 [sys.catalog]: This master's current role is: LEADER
W20260929 17:57:29.861954 22901 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 895df0fd36404c8a95cd03fb51ae877a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260929 17:57:29.864496 22853 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "17eeece19fda4a509b8f3bb514630d83" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "895df0fd36404c8a95cd03fb51ae877a"
I20260929 17:57:29.873113 22853 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 895df0fd36404c8a95cd03fb51ae877a [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:57:29.873420 22933 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 17:57:29.874480 22853 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 895df0fd36404c8a95cd03fb51ae877a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17eeece19fda4a509b8f3bb514630d83 in term 1.
W20260929 17:57:29.874706 22901 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 17:57:29.876855 22933 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 17:57:29.884713 22733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:29.875173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47216 (local address 127.0.98.62:43861)
0929 17:57:29.875305 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0929 17:57:29.875310 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:57:29.875716 (+ 406us) server_negotiation.cc:407] Connection header received
0929 17:57:29.875917 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:57:29.875920 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:57:29.875970 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:57:29.876022 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:57:29.877095 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:29.877812 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:29.878991 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:29.879301 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:29.879369 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:29.881199 (+ 1830us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:57:29.881203 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:57:29.881206 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:57:29.881233 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:57:29.882908 (+ 1675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:57:29.883341 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:57:29.883344 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:57:29.883346 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:57:29.883430 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:57:29.883726 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:57:29.883729 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:57:29.883731 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:57:29.883949 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:57:29.884053 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0929 17:57:29.884417 (+ 364us) server_negotiation.cc:299] Negotiation successful
0929 17:57:29.884533 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 17:57:29.886444 22885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:29.875860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33518 (local address 127.0.98.60:39169)
0929 17:57:29.875968 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0929 17:57:29.875972 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:57:29.877318 (+ 1346us) server_negotiation.cc:407] Connection header received
0929 17:57:29.877473 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:57:29.877476 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:57:29.877520 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:57:29.877564 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:57:29.878475 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:29.879223 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:29.880260 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:29.880640 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:29.880767 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:29.882770 (+ 2003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:57:29.882775 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:57:29.882778 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:57:29.882809 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:57:29.884545 (+ 1736us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:57:29.885084 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:57:29.885088 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:57:29.885090 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:57:29.885184 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:57:29.885531 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:57:29.885534 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:57:29.885536 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:57:29.885750 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:57:29.885878 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0929 17:57:29.886191 (+ 313us) server_negotiation.cc:299] Negotiation successful
0929 17:57:29.886307 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 17:57:29.889382 22809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:29.876593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47220 (local address 127.0.98.61:35953)
0929 17:57:29.876714 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0929 17:57:29.876718 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:57:29.880418 (+ 3700us) server_negotiation.cc:407] Connection header received
0929 17:57:29.880593 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:57:29.880596 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:57:29.880653 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:57:29.880706 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:57:29.881506 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:29.882234 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:29.883314 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:29.883623 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:29.883696 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:29.885333 (+ 1637us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:57:29.885337 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:57:29.885340 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:57:29.885366 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:57:29.886969 (+ 1603us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:57:29.888031 (+ 1062us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:57:29.888035 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:57:29.888037 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:57:29.888130 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:57:29.888468 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:57:29.888472 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:57:29.888473 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:57:29.888697 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:57:29.888845 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0929 17:57:29.889126 (+ 281us) server_negotiation.cc:299] Negotiation successful
0929 17:57:29.889239 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 17:57:29.897608 22853 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 895df0fd36404c8a95cd03fb51ae877a [term 1 FOLLOWER]: Refusing update from remote peer 17eeece19fda4a509b8f3bb514630d83: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 17:57:29.897609 22778 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 919eecd744334eb687bad6598efa11ed [term 1 FOLLOWER]: Refusing update from remote peer 17eeece19fda4a509b8f3bb514630d83: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 17:57:29.898696 22895 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 17eeece19fda4a509b8f3bb514630d83 [LEADER]: Connected to new peer: Peer: permanent_uuid: "919eecd744334eb687bad6598efa11ed" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 17:57:29.899120 22895 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 17eeece19fda4a509b8f3bb514630d83 [LEADER]: Connected to new peer: Peer: permanent_uuid: "895df0fd36404c8a95cd03fb51ae877a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39169 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 17:57:29.902889 22889 sys_catalog.cc:455] T 00000000000000000000000000000000 P 895df0fd36404c8a95cd03fb51ae877a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 17eeece19fda4a509b8f3bb514630d83. Latest consensus state: current_term: 1 leader_uuid: "17eeece19fda4a509b8f3bb514630d83" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17eeece19fda4a509b8f3bb514630d83" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43861 } } peers { permanent_uuid: "919eecd744334eb687bad6598efa11ed" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35953 } } peers { permanent_uuid: "895df0fd36404c8a95cd03fb51ae877a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39169 } } }
I20260929 17:57:29.903029 22889 sys_catalog.cc:458] T 00000000000000000000000000000000 P 895df0fd36404c8a95cd03fb51ae877a [sys.catalog]: This master's current role is: FOLLOWER
I20260929 17:57:29.903245 22902 sys_catalog.cc:455] T 00000000000000000000000000000000 P 919eecd744334eb687bad6598efa11ed [sys.catalog]: SysCatalogTable state changed. Reason: New leader 17eeece19fda4a509b8f3bb514630d83. Latest consensus state: current_term: 1 leader_uuid: "17eeece19fda4a509b8f3bb514630d83" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17eeece19fda4a509b8f3bb514630d83" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43861 } } peers { permanent_uuid: "919eecd744334eb687bad6598efa11ed" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35953 } } peers { permanent_uuid: "895df0fd36404c8a95cd03fb51ae877a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39169 } } }
I20260929 17:57:29.903388 22902 sys_catalog.cc:458] T 00000000000000000000000000000000 P 919eecd744334eb687bad6598efa11ed [sys.catalog]: This master's current role is: FOLLOWER
I20260929 17:57:29.919970 22939 sys_catalog.cc:455] T 00000000000000000000000000000000 P 17eeece19fda4a509b8f3bb514630d83 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "17eeece19fda4a509b8f3bb514630d83" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17eeece19fda4a509b8f3bb514630d83" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43861 } } peers { permanent_uuid: "919eecd744334eb687bad6598efa11ed" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35953 } } peers { permanent_uuid: "895df0fd36404c8a95cd03fb51ae877a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39169 } } }
I20260929 17:57:29.920136 22939 sys_catalog.cc:458] T 00000000000000000000000000000000 P 17eeece19fda4a509b8f3bb514630d83 [sys.catalog]: This master's current role is: LEADER
I20260929 17:57:29.920249 22938 mvcc.cc:205] Tried to move back new op lower bound from 7334726245969563648 to 7334726245874642944. Current Snapshot: MvccSnapshot[applied={T|T < 7334726245969563648}]
I20260929 17:57:29.921739 22889 sys_catalog.cc:455] T 00000000000000000000000000000000 P 895df0fd36404c8a95cd03fb51ae877a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "17eeece19fda4a509b8f3bb514630d83" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17eeece19fda4a509b8f3bb514630d83" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43861 } } peers { permanent_uuid: "919eecd744334eb687bad6598efa11ed" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35953 } } peers { permanent_uuid: "895df0fd36404c8a95cd03fb51ae877a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39169 } } }
I20260929 17:57:29.921862 22889 sys_catalog.cc:458] T 00000000000000000000000000000000 P 895df0fd36404c8a95cd03fb51ae877a [sys.catalog]: This master's current role is: FOLLOWER
I20260929 17:57:29.925820 22895 sys_catalog.cc:455] T 00000000000000000000000000000000 P 17eeece19fda4a509b8f3bb514630d83 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "17eeece19fda4a509b8f3bb514630d83" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17eeece19fda4a509b8f3bb514630d83" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43861 } } peers { permanent_uuid: "919eecd744334eb687bad6598efa11ed" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35953 } } peers { permanent_uuid: "895df0fd36404c8a95cd03fb51ae877a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39169 } } }
I20260929 17:57:29.925942 22895 sys_catalog.cc:458] T 00000000000000000000000000000000 P 17eeece19fda4a509b8f3bb514630d83 [sys.catalog]: This master's current role is: LEADER
I20260929 17:57:29.927045 22902 sys_catalog.cc:455] T 00000000000000000000000000000000 P 919eecd744334eb687bad6598efa11ed [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "17eeece19fda4a509b8f3bb514630d83" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17eeece19fda4a509b8f3bb514630d83" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43861 } } peers { permanent_uuid: "919eecd744334eb687bad6598efa11ed" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35953 } } peers { permanent_uuid: "895df0fd36404c8a95cd03fb51ae877a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39169 } } }
I20260929 17:57:29.927198 22902 sys_catalog.cc:458] T 00000000000000000000000000000000 P 919eecd744334eb687bad6598efa11ed [sys.catalog]: This master's current role is: FOLLOWER
I20260929 17:57:29.927523 22940 mvcc.cc:205] Tried to move back new op lower bound from 7334726245969563648 to 7334726245874642944. Current Snapshot: MvccSnapshot[applied={T|T < 7334726245969563648}]
I20260929 17:57:29.927675 22933 catalog_manager.cc:1347] Generated new cluster ID: 179c12f71ba64cac9c22b107812bcbbf
I20260929 17:57:29.927745 22933 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 17:57:29.928045 22941 mvcc.cc:205] Tried to move back new op lower bound from 7334726245969563648 to 7334726245874642944. Current Snapshot: MvccSnapshot[applied={T|T < 7334726245969563648}]
I20260929 17:57:29.965287 22933 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 17:57:29.966718 22933 catalog_manager.cc:1504] Loading token signing keys...
I20260929 17:57:29.995445 22933 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 17eeece19fda4a509b8f3bb514630d83: Generated new TSK 0
I20260929 17:57:29.996433 22933 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 17:57:30.033282 22809 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:57:30.030741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47222 (local address 127.0.98.61:35953)
0929 17:57:30.030885 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0929 17:57:30.030890 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:57:30.032067 (+ 1177us) server_negotiation.cc:407] Connection header received
0929 17:57:30.032287 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:57:30.032291 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:57:30.032348 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:57:30.032419 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:57:30.033129 (+ 710us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47222: BlockingRecv error: recv got EOF from 127.0.0.1:47222 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
W20260929 17:57:30.035027 22885 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:57:30.031714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33532 (local address 127.0.98.60:39169)
0929 17:57:30.031834 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0929 17:57:30.031839 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:57:30.033880 (+ 2041us) server_negotiation.cc:407] Connection header received
0929 17:57:30.034152 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:57:30.034155 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:57:30.034212 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:57:30.034267 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:57:30.034891 (+ 624us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33532: BlockingRecv error: recv got EOF from 127.0.0.1:33532 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
W20260929 17:57:30.036574 22733 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:57:30.035506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47228 (local address 127.0.98.62:43861)
0929 17:57:30.035623 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0929 17:57:30.035628 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:57:30.035644 (+ 16us) server_negotiation.cc:407] Connection header received
0929 17:57:30.035789 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:57:30.035793 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:57:30.035844 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:57:30.035896 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:57:30.036444 (+ 548us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47228: BlockingRecv error: recv got EOF from 127.0.0.1:47228 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260929 17:57:30.041666 22809 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:57:30.039961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47230 (local address 127.0.98.61:35953)
0929 17:57:30.040086 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0929 17:57:30.040091 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:57:30.040569 (+ 478us) server_negotiation.cc:407] Connection header received
0929 17:57:30.040809 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:57:30.040812 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:57:30.040904 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:57:30.040960 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:57:30.041557 (+ 597us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47230: BlockingRecv error: recv got EOF from 127.0.0.1:47230 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260929 17:57:30.046938 22809 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:57:30.045309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47240 (local address 127.0.98.61:35953)
0929 17:57:30.045428 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0929 17:57:30.045432 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:57:30.045982 (+ 550us) server_negotiation.cc:407] Connection header received
0929 17:57:30.046179 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:57:30.046183 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:57:30.046242 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:57:30.046303 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:57:30.046831 (+ 528us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47240: BlockingRecv error: recv got EOF from 127.0.0.1:47240 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260929 17:57:30.052778 22809 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:57:30.050923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47248 (local address 127.0.98.61:35953)
0929 17:57:30.051037 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0929 17:57:30.051042 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:57:30.051665 (+ 623us) server_negotiation.cc:407] Connection header received
0929 17:57:30.051910 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:57:30.051913 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:57:30.051969 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:57:30.052026 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:57:30.052644 (+ 618us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47248: BlockingRecv error: recv got EOF from 127.0.0.1:47248 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260929 17:57:30.066519 22809 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:57:30.064397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47262 (local address 127.0.98.61:35953)
0929 17:57:30.064584 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0929 17:57:30.064590 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 17:57:30.065387 (+ 797us) server_negotiation.cc:407] Connection header received
0929 17:57:30.065601 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:57:30.065605 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:57:30.065663 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:57:30.065722 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:57:30.066385 (+ 663us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47262: BlockingRecv error: recv got EOF from 127.0.0.1:47262 (error 108)
Metrics: {"server-negotiator.queue_time_us":91}
W20260929 17:57:30.078586 22809 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:57:30.077244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47274 (local address 127.0.98.61:35953)
0929 17:57:30.077361 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0929 17:57:30.077367 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 17:57:30.077761 (+ 394us) server_negotiation.cc:407] Connection header received
0929 17:57:30.077886 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:57:30.077890 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:57:30.077942 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:57:30.078008 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:57:30.078466 (+ 458us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47274: BlockingRecv error: recv got EOF from 127.0.0.1:47274 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
W20260929 17:57:30.097908 22809 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:57:30.096690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47282 (local address 127.0.98.61:35953)
0929 17:57:30.096816 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0929 17:57:30.096820 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:57:30.097068 (+ 248us) server_negotiation.cc:407] Connection header received
0929 17:57:30.097223 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:57:30.097226 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:57:30.097275 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:57:30.097343 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:57:30.097793 (+ 450us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47282: BlockingRecv error: recv got EOF from 127.0.0.1:47282 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260929 17:57:30.133144 22809 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:57:30.132009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47298 (local address 127.0.98.61:35953)
0929 17:57:30.132150 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0929 17:57:30.132155 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:57:30.132267 (+ 112us) server_negotiation.cc:407] Connection header received
0929 17:57:30.132421 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:57:30.132424 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:57:30.132472 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:57:30.132542 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:57:30.133034 (+ 492us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47298: BlockingRecv error: recv got EOF from 127.0.0.1:47298 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260929 17:57:30.200613 22809 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:57:30.199376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47302 (local address 127.0.98.61:35953)
0929 17:57:30.199529 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0929 17:57:30.199534 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:57:30.199684 (+ 150us) server_negotiation.cc:407] Connection header received
0929 17:57:30.199809 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:57:30.199812 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:57:30.199903 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:57:30.199979 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:57:30.200490 (+ 511us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47302: BlockingRecv error: recv got EOF from 127.0.0.1:47302 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
W20260929 17:57:30.332054 22809 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:57:30.330932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47306 (local address 127.0.98.61:35953)
0929 17:57:30.331084 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0929 17:57:30.331088 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:57:30.331236 (+ 148us) server_negotiation.cc:407] Connection header received
0929 17:57:30.331395 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:57:30.331398 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:57:30.331450 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:57:30.331515 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:57:30.331930 (+ 415us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47306: BlockingRecv error: recv got EOF from 127.0.0.1:47306 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260929 17:57:30.591432 22809 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:57:30.590369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47318 (local address 127.0.98.61:35953)
0929 17:57:30.590533 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0929 17:57:30.590536 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 17:57:30.590657 (+ 121us) server_negotiation.cc:407] Connection header received
0929 17:57:30.590807 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:57:30.590810 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:57:30.590859 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:57:30.590920 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:57:30.591311 (+ 391us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47318: BlockingRecv error: recv got EOF from 127.0.0.1:47318 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
I20260929 17:57:30.869787 22925 catalog_manager.cc:1259] Loaded cluster ID: 179c12f71ba64cac9c22b107812bcbbf
I20260929 17:57:30.869882 22925 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 919eecd744334eb687bad6598efa11ed: loading cluster ID for follower catalog manager: success
I20260929 17:57:30.871663 22925 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 919eecd744334eb687bad6598efa11ed: acquiring CA information for follower catalog manager: success
I20260929 17:57:30.872889 22925 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 919eecd744334eb687bad6598efa11ed: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 17:57:30.875890 22901 catalog_manager.cc:1259] Loaded cluster ID: 179c12f71ba64cac9c22b107812bcbbf
I20260929 17:57:30.875955 22901 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 895df0fd36404c8a95cd03fb51ae877a: loading cluster ID for follower catalog manager: success
I20260929 17:57:30.877606 22901 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 895df0fd36404c8a95cd03fb51ae877a: acquiring CA information for follower catalog manager: success
I20260929 17:57:30.878774 22901 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 895df0fd36404c8a95cd03fb51ae877a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 17:57:31.109730 23011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:31.105588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47330 (local address 127.0.98.61:35953)
0929 17:57:31.105837 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0929 17:57:31.105842 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:57:31.105901 (+ 59us) server_negotiation.cc:407] Connection header received
0929 17:57:31.106077 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:57:31.106080 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:57:31.106226 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:57:31.106325 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 17:57:31.106602 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:31.107174 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:31.108095 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:31.108346 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:31.108437 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:31.108775 (+ 338us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 17:57:31.109290 (+ 515us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 17:57:31.109406 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0929 17:57:31.109488 (+ 82us) server_negotiation.cc:299] Negotiation successful
0929 17:57:31.109557 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":172,"thread_start_us":81,"threads_started":1}
I20260929 17:57:31.116616 23017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:31.112898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33540 (local address 127.0.98.60:39169)
0929 17:57:31.113131 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0929 17:57:31.113135 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:57:31.113148 (+ 13us) server_negotiation.cc:407] Connection header received
0929 17:57:31.113292 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:57:31.113295 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:57:31.113429 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:57:31.113532 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 17:57:31.113707 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:31.114233 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:31.115032 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:31.115286 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:31.115368 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:31.115605 (+ 237us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 17:57:31.116198 (+ 593us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 17:57:31.116307 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0929 17:57:31.116424 (+ 117us) server_negotiation.cc:299] Negotiation successful
0929 17:57:31.116479 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":62,"threads_started":1}
I20260929 17:57:31.124123 23011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:31.119886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47332 (local address 127.0.98.61:35953)
0929 17:57:31.120256 (+ 370us) server_negotiation.cc:206] Beginning negotiation
0929 17:57:31.120260 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:57:31.120534 (+ 274us) server_negotiation.cc:407] Connection header received
0929 17:57:31.120645 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:57:31.120648 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:57:31.120705 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:57:31.120774 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 17:57:31.121039 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:31.121558 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:31.122852 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:31.123190 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:31.123283 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:31.123592 (+ 309us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 17:57:31.123695 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 17:57:31.123760 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0929 17:57:31.123901 (+ 141us) server_negotiation.cc:299] Negotiation successful
0929 17:57:31.123965 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":267}
I20260929 17:57:31.124325 23017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:31.120343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33554 (local address 127.0.98.60:39169)
0929 17:57:31.120466 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0929 17:57:31.120472 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 17:57:31.120486 (+ 14us) server_negotiation.cc:407] Connection header received
0929 17:57:31.120535 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:57:31.120538 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:57:31.120584 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:57:31.120648 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 17:57:31.120919 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:31.121723 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:31.123348 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:31.123583 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:31.123680 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:31.123872 (+ 192us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 17:57:31.123945 (+ 73us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 17:57:31.124005 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0929 17:57:31.124141 (+ 136us) server_negotiation.cc:299] Negotiation successful
0929 17:57:31.124196 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260929 17:57:31.124907 23024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:31.119892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47234 (local address 127.0.98.62:43861)
0929 17:57:31.120295 (+ 403us) server_negotiation.cc:206] Beginning negotiation
0929 17:57:31.120299 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:57:31.120313 (+ 14us) server_negotiation.cc:407] Connection header received
0929 17:57:31.120416 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:57:31.120419 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:57:31.120571 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:57:31.121035 (+ 464us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 17:57:31.121181 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:31.121903 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:31.122810 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:31.123156 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:31.123262 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:31.123850 (+ 588us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 17:57:31.124227 (+ 377us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 17:57:31.124669 (+ 442us) server_negotiation.cc:1035] Waiting for connection context
0929 17:57:31.124707 (+ 38us) server_negotiation.cc:299] Negotiation successful
0929 17:57:31.124771 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":332,"thread_start_us":78,"threads_started":1}
I20260929 17:57:31.132789 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskup3Soe/build/debug/bin/kudu with pid 22661
I20260929 17:57:31.138289 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskup3Soe/build/debug/bin/kudu with pid 22736
I20260929 17:57:31.143492 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskup3Soe/build/debug/bin/kudu with pid 22810
2026-09-29T17:57:31Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (3032 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790704375975685-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 17:57:31 dist-test-slave-53nh krb5kdc[23028](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 17:57:31 dist-test-slave-53nh krb5kdc[23028](info): set up 2 sockets
Sep 29 17:57:31 dist-test-slave-53nh krb5kdc[23028](info): commencing operation
krb5kdc: starting...
W20260929 17:57:32.101258 392 mini_kdc.cc:121] Time spent starting KDC: real 0.938s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790704375975685-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790704375975685-392-0/krb5kdc/test-user.keytab.
Sep 29 17:57:32 dist-test-slave-53nh krb5kdc[23028](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704652, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T17:57:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T17:57:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790704375975685-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790704375975685-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 17:57:32.218094 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskup3Soe/build/debug/bin/kudu
/tmp/dist-test-taskup3Soe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790704375975685-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790704375975685-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790704375975685-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790704375975685-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44477
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790704375975685-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44477,127.0.98.61:33593,127.0.98.60:40643
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790704375975685-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790704375975685-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790704375975685-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 17:57:32.330883 23041 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 17:57:32.331207 23041 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:57:32.331248 23041 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:57:32.331314 23041 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 17:57:32.335465 23041 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 17:57:32.335546 23041 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:57:32.335575 23041 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:57:32.335595 23041 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 17:57:32.335615 23041 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 17:57:32.341219 23041 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790704375975685-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790704375975685-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44477,127.0.98.61:33593,127.0.98.60:40643
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790704375975685-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44477
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790704375975685-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23041
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790704375975685-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:57:32.342406 23041 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:57:32.343495 23041 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:57:32.353132 23047 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:57:32.353130 23046 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:57:32.353147 23049 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:57:32.353331 23041 server_base.cc:1034] running on GCE node
I20260929 17:57:32.354041 23041 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:57:32.355171 23041 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:57:32.356361 23041 hybrid_clock.cc:648] HybridClock initialized: now 1790704652356343 us; error 38 us; skew 500 ppm
Sep 29 17:57:32 dist-test-slave-53nh krb5kdc[23028](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704652, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:57:32.359278 23041 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 17:57:32.360265 23041 webserver.cc:460] Webserver started at http://127.0.98.62:36955/ using document root <none> and password file <none>
I20260929 17:57:32.360852 23041 fs_manager.cc:359] Metadata directory not provided
I20260929 17:57:32.360921 23041 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:57:32.361181 23041 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:57:32.363356 23041 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790704375975685-392-0/minicluster-data/master-0/data/instance:
uuid: "eb7d936e30da40d2b5b05cb87da5ee13"
format_stamp: "Formatted at 2026-09-29 17:57:32 on dist-test-slave-53nh"
server_key: "eeb9d52e2cbb8b934852acb86c8542d5"
server_key_iv: "0cd01e2abc92119a94f71a700fc882ac"
server_key_version: "encryptionkey@0"
I20260929 17:57:32.363933 23041 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790704375975685-392-0/minicluster-data/master-0/wal/instance:
uuid: "eb7d936e30da40d2b5b05cb87da5ee13"
format_stamp: "Formatted at 2026-09-29 17:57:32 on dist-test-slave-53nh"
server_key: "eeb9d52e2cbb8b934852acb86c8542d5"
server_key_iv: "0cd01e2abc92119a94f71a700fc882ac"
server_key_version: "encryptionkey@0"
I20260929 17:57:32.367839 23041 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 17:57:32.370363 23056 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:57:32.371622 23041 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 17:57:32.371744 23041 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790704375975685-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790704375975685-392-0/minicluster-data/master-0/wal
uuid: "eb7d936e30da40d2b5b05cb87da5ee13"
format_stamp: "Formatted at 2026-09-29 17:57:32 on dist-test-slave-53nh"
server_key: "eeb9d52e2cbb8b934852acb86c8542d5"
server_key_iv: "0cd01e2abc92119a94f71a700fc882ac"
server_key_version: "encryptionkey@0"
I20260929 17:57:32.371855 23041 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790704375975685-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790704375975685-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790704375975685-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:57:32.398120 23041 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:57:32.401297 23041 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:57:32.401516 23041 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:57:32.408552 23041 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44477
I20260929 17:57:32.408826 23108 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44477 every 8 connection(s)
I20260929 17:57:32.409569 23041 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790704375975685-392-0/minicluster-data/master-0/data/info.pb
I20260929 17:57:32.412299 23109 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:57:32.414728 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskup3Soe/build/debug/bin/kudu as pid 23041
I20260929 17:57:32.414862 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790704375975685-392-0/minicluster-data/master-0/wal/instance
I20260929 17:57:32.415155 392 external_mini_cluster.cc:1428] Setting key c493ff040691a1b96278869246af68ff
I20260929 17:57:32.416620 23109 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44477 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Sep 29 17:57:32 dist-test-slave-53nh krb5kdc[23028](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704652, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 17:57:32.432699 23113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:32.419855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59192 (local address 127.0.98.62:44477)
0929 17:57:32.420179 (+ 324us) server_negotiation.cc:206] Beginning negotiation
0929 17:57:32.420190 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 17:57:32.420239 (+ 49us) server_negotiation.cc:407] Connection header received
0929 17:57:32.420733 (+ 494us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:57:32.420741 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:57:32.421046 (+ 305us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:57:32.421267 (+ 221us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:57:32.422055 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:32.423298 (+ 1243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:32.424347 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:32.424737 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:32.424855 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:32.427585 (+ 2730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:57:32.427591 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:57:32.427603 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:57:32.427627 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:57:32.429405 (+ 1778us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:57:32.429883 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:57:32.429889 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:57:32.429890 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:57:32.430065 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:57:32.430406 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:57:32.430410 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:57:32.430411 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:57:32.430846 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:57:32.430993 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0929 17:57:32.431361 (+ 368us) server_negotiation.cc:299] Negotiation successful
0929 17:57:32.431603 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":52,"server-negotiator.queue_time_us":211,"thread_start_us":94,"threads_started":1}
I20260929 17:57:32.432699 23112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:32.418938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44477 (local address 127.0.0.1:59192)
0929 17:57:32.419438 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0929 17:57:32.419490 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0929 17:57:32.420418 (+ 928us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:57:32.421257 (+ 839us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:57:32.421261 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:57:32.421583 (+ 322us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:57:32.421906 (+ 323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:57:32.421921 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:57:32.423410 (+ 1489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:57:32.423412 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:57:32.424245 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:57:32.424255 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:57:32.424886 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:57:32.424889 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:57:32.424968 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:32.425665 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:57:32.425681 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:57:32.427414 (+ 1733us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:57:32.429520 (+ 2106us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:57:32.429529 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:57:32.429542 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:57:32.429778 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:57:32.430154 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:57:32.430157 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:57:32.430159 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:57:32.430310 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:57:32.430960 (+ 650us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:57:32.430967 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:57:32.431116 (+ 149us) client_negotiation.cc:769] Sending connection context
0929 17:57:32.431276 (+ 160us) client_negotiation.cc:240] Negotiation successful
0929 17:57:32.431660 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292,"mutex_wait_us":116,"thread_start_us":136,"threads_started":1}
I20260929 17:57:32.435333 23109 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33593 } has no permanent_uuid. Determining permanent_uuid...
I20260929 17:57:32.436071 23112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:32.435794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33593 (local address 127.0.0.1:44656)
0929 17:57:32.435917 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0929 17:57:32.435999 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260929 17:57:32.436285 23060 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33593: connect: Connection refused (error 111)
I20260929 17:57:32.437160 23112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:32.436869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33593 (local address 127.0.0.1:44660)
0929 17:57:32.436976 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0929 17:57:32.437093 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260929 17:57:32.437294 23109 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:33593: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33593: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790704375975685-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790704375975685-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260929 17:57:32.443328 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskup3Soe/build/debug/bin/kudu
/tmp/dist-test-taskup3Soe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790704375975685-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790704375975685-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790704375975685-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790704375975685-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:33593
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790704375975685-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44477,127.0.98.61:33593,127.0.98.60:40643
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790704375975685-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790704375975685-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790704375975685-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 17:57:32.486572 23109 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33593 } attempt: 1
I20260929 17:57:32.487473 23112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:32.487154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33593 (local address 127.0.0.1:44664)
0929 17:57:32.487320 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0929 17:57:32.487387 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260929 17:57:32.488283 23112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:32.488067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33593 (local address 127.0.0.1:44674)
0929 17:57:32.488170 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0929 17:57:32.488220 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260929 17:57:32.488415 23109 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:33593: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33593: connect: Connection refused (error 111)
W20260929 17:57:32.554574 23116 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 17:57:32.554908 23116 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:57:32.554950 23116 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:57:32.555015 23116 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260929 17:57:32.556589 23109 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33593 } attempt: 2
I20260929 17:57:32.557480 23112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:32.557184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33593 (local address 127.0.0.1:44688)
0929 17:57:32.557336 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0929 17:57:32.557402 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260929 17:57:32.558164 23112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:32.557953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33593 (local address 127.0.0.1:44702)
0929 17:57:32.558050 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0929 17:57:32.558096 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260929 17:57:32.558303 23109 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:33593: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33593: connect: Connection refused (error 111)
W20260929 17:57:32.559546 23116 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 17:57:32.559610 23116 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:57:32.559630 23116 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:57:32.559646 23116 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 17:57:32.559677 23116 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 17:57:32.565456 23116 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790704375975685-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790704375975685-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44477,127.0.98.61:33593,127.0.98.60:40643
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790704375975685-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:33593
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790704375975685-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23116
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790704375975685-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:57:32.566520 23116 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:57:32.567548 23116 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:57:32.575904 23121 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:57:32.575944 23124 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:57:32.575917 23122 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:57:32.576404 23116 server_base.cc:1034] running on GCE node
I20260929 17:57:32.576972 23116 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:57:32.578133 23116 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:57:32.579291 23116 hybrid_clock.cc:648] HybridClock initialized: now 1790704652579278 us; error 33 us; skew 500 ppm
Sep 29 17:57:32 dist-test-slave-53nh krb5kdc[23028](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704652, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:57:32.581969 23116 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260929 17:57:32.582876 23116 webserver.cc:460] Webserver started at http://127.0.98.61:45059/ using document root <none> and password file <none>
I20260929 17:57:32.583395 23116 fs_manager.cc:359] Metadata directory not provided
I20260929 17:57:32.583457 23116 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:57:32.583659 23116 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:57:32.585734 23116 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790704375975685-392-0/minicluster-data/master-1/data/instance:
uuid: "ac2d4de906974b579e10066188cb24a6"
format_stamp: "Formatted at 2026-09-29 17:57:32 on dist-test-slave-53nh"
server_key: "24f5f1ae0c4e41195487089a3168a6ac"
server_key_iv: "7b8c22864030517b65c7b5459cb16845"
server_key_version: "encryptionkey@0"
I20260929 17:57:32.586293 23116 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790704375975685-392-0/minicluster-data/master-1/wal/instance:
uuid: "ac2d4de906974b579e10066188cb24a6"
format_stamp: "Formatted at 2026-09-29 17:57:32 on dist-test-slave-53nh"
server_key: "24f5f1ae0c4e41195487089a3168a6ac"
server_key_iv: "7b8c22864030517b65c7b5459cb16845"
server_key_version: "encryptionkey@0"
I20260929 17:57:32.590305 23116 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 17:57:32.592990 23131 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:57:32.594321 23116 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 17:57:32.594442 23116 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790704375975685-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790704375975685-392-0/minicluster-data/master-1/wal
uuid: "ac2d4de906974b579e10066188cb24a6"
format_stamp: "Formatted at 2026-09-29 17:57:32 on dist-test-slave-53nh"
server_key: "24f5f1ae0c4e41195487089a3168a6ac"
server_key_iv: "7b8c22864030517b65c7b5459cb16845"
server_key_version: "encryptionkey@0"
I20260929 17:57:32.594549 23116 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790704375975685-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790704375975685-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790704375975685-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:57:32.609369 23116 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:57:32.612448 23116 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:57:32.612645 23116 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:57:32.619593 23116 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:33593
I20260929 17:57:32.619861 23183 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:33593 every 8 connection(s)
I20260929 17:57:32.620556 23116 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790704375975685-392-0/minicluster-data/master-1/data/info.pb
I20260929 17:57:32.623263 23184 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:57:32.627120 23184 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44477 } has no permanent_uuid. Determining permanent_uuid...
I20260929 17:57:32.629508 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskup3Soe/build/debug/bin/kudu as pid 23116
I20260929 17:57:32.629626 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790704375975685-392-0/minicluster-data/master-1/wal/instance
I20260929 17:57:32.629891 392 external_mini_cluster.cc:1428] Setting key 0edfdb8426646b337ead22b01b428c86
Sep 29 17:57:32 dist-test-slave-53nh krb5kdc[23028](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704652, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
I20260929 17:57:32.641916 23113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:32.629173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59206 (local address 127.0.98.62:44477)
0929 17:57:32.629330 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0929 17:57:32.629335 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:57:32.630341 (+ 1006us) server_negotiation.cc:407] Connection header received
0929 17:57:32.631733 (+ 1392us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:57:32.631737 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:57:32.631800 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:57:32.631899 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:57:32.633275 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:32.633851 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:32.635050 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:32.635298 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:32.635395 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:32.638188 (+ 2793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:57:32.638191 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:57:32.638193 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:57:32.638221 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:57:32.639615 (+ 1394us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:57:32.640189 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:57:32.640194 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:57:32.640195 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:57:32.640292 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:57:32.640664 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:57:32.640668 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:57:32.640669 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:57:32.640865 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:57:32.640953 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0929 17:57:32.641589 (+ 636us) server_negotiation.cc:299] Negotiation successful
0929 17:57:32.641748 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 17:57:32.642714 23186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:32.629583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44477 (local address 127.0.0.1:59206)
0929 17:57:32.630149 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0929 17:57:32.630218 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0929 17:57:32.631430 (+ 1212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:57:32.632055 (+ 625us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:57:32.632061 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:57:32.632424 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:57:32.633092 (+ 668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:57:32.633115 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:57:32.634019 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:57:32.634022 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:57:32.634918 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:57:32.634928 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:57:32.635462 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:57:32.635466 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:57:32.635640 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:32.636310 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:57:32.636328 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:57:32.637976 (+ 1648us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:57:32.639757 (+ 1781us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:57:32.639762 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:57:32.639776 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:57:32.640047 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:57:32.640386 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:57:32.640389 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:57:32.640391 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:57:32.640548 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:57:32.640969 (+ 421us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:57:32.640977 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:57:32.641323 (+ 346us) client_negotiation.cc:769] Sending connection context
0929 17:57:32.641496 (+ 173us) client_negotiation.cc:240] Negotiation successful
0929 17:57:32.641859 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":143,"threads_started":1}
I20260929 17:57:32.644068 23184 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33593 } has no permanent_uuid. Determining permanent_uuid...
I20260929 17:57:32.649448 23109 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33593 } attempt: 3
Sep 29 17:57:32 dist-test-slave-53nh krb5kdc[23028](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704652, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260929 17:57:32.654557 23186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:32.644590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33593 (local address 127.0.0.1:44716)
0929 17:57:32.644705 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0929 17:57:32.644725 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 17:57:32.644805 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:57:32.645464 (+ 659us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:57:32.645467 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:57:32.645608 (+ 141us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:57:32.645705 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:57:32.645710 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:57:32.647115 (+ 1405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:57:32.647118 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:57:32.647652 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:57:32.647659 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:57:32.648346 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:57:32.648349 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:57:32.648429 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:32.648897 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:57:32.648908 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:57:32.650912 (+ 2004us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:57:32.652966 (+ 2054us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:57:32.652969 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:57:32.652971 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:57:32.653214 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:57:32.653482 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:57:32.653485 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:57:32.653486 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:57:32.653574 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:57:32.654058 (+ 484us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:57:32.654062 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:57:32.654154 (+ 92us) client_negotiation.cc:769] Sending connection context
0929 17:57:32.654256 (+ 102us) client_negotiation.cc:240] Negotiation successful
0929 17:57:32.654356 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
Sep 29 17:57:32 dist-test-slave-53nh krb5kdc[23028](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704652, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260929 17:57:32.655084 23189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:32.644686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44716 (local address 127.0.98.61:33593)
0929 17:57:32.644986 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0929 17:57:32.645020 (+ 34us) server_negotiation.cc:399] Waiting for connection header
0929 17:57:32.645044 (+ 24us) server_negotiation.cc:407] Connection header received
0929 17:57:32.645090 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:57:32.645106 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:57:32.645318 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:57:32.645461 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:57:32.645818 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:32.646996 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:32.647770 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:32.648219 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:32.648304 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:32.651026 (+ 2722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:57:32.651038 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:57:32.651055 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:57:32.651079 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:57:32.652865 (+ 1786us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:57:32.653315 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:57:32.653319 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:57:32.653321 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:57:32.653388 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:57:32.653655 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:57:32.653659 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:57:32.653661 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:57:32.653951 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:57:32.654075 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0929 17:57:32.654356 (+ 281us) server_negotiation.cc:299] Negotiation successful
0929 17:57:32.654502 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":194,"thread_start_us":98,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790704375975685-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790704375975685-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260929 17:57:32.656862 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskup3Soe/build/debug/bin/kudu
/tmp/dist-test-taskup3Soe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790704375975685-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790704375975685-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790704375975685-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790704375975685-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:40643
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790704375975685-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44477,127.0.98.61:33593,127.0.98.60:40643
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790704375975685-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790704375975685-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790704375975685-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 17:57:32.657042 23184 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40643 } has no permanent_uuid. Determining permanent_uuid...
I20260929 17:57:32.657737 23186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:32.657473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40643 (local address 127.0.0.1:56194)
0929 17:57:32.657615 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0929 17:57:32.657681 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40643: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
W20260929 17:57:32.657918 23133 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40643: connect: Connection refused (error 111)
I20260929 17:57:32.658470 23112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:32.649852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33593 (local address 127.0.0.1:44728)
0929 17:57:32.650024 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0929 17:57:32.650040 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 17:57:32.650132 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:57:32.650454 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:57:32.650456 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:57:32.650612 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:57:32.650721 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:57:32.650728 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:57:32.651773 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:57:32.651776 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:57:32.652547 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:57:32.652558 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:57:32.653186 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:57:32.653190 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:57:32.653300 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:32.653732 (+ 432us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:57:32.653743 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:57:32.655149 (+ 1406us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:57:32.656971 (+ 1822us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:57:32.656974 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:57:32.656976 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:57:32.657240 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:57:32.657540 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:57:32.657543 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:57:32.657545 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:57:32.657630 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:57:32.658060 (+ 430us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:57:32.658063 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:57:32.658175 (+ 112us) client_negotiation.cc:769] Sending connection context
0929 17:57:32.658253 (+ 78us) client_negotiation.cc:240] Negotiation successful
0929 17:57:32.658357 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260929 17:57:32.658756 23186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:32.658550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40643 (local address 127.0.0.1:56202)
0929 17:57:32.658658 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0929 17:57:32.658702 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40643: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260929 17:57:32.658885 23184 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:40643: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40643: connect: Connection refused (error 111)
I20260929 17:57:32.659022 23190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:32.649927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44728 (local address 127.0.98.61:33593)
0929 17:57:32.650272 (+ 345us) server_negotiation.cc:206] Beginning negotiation
0929 17:57:32.650278 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 17:57:32.650288 (+ 10us) server_negotiation.cc:407] Connection header received
0929 17:57:32.650313 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:57:32.650315 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:57:32.650348 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:57:32.650418 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:57:32.650870 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:32.651655 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:32.652672 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:32.653086 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:32.653163 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:32.655300 (+ 2137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:57:32.655304 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:57:32.655306 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:57:32.655327 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:57:32.656865 (+ 1538us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:57:32.657348 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:57:32.657351 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:57:32.657353 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:57:32.657433 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:57:32.657729 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:57:32.657731 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:57:32.657733 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:57:32.657942 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:57:32.658637 (+ 695us) server_negotiation.cc:1035] Waiting for connection context
0929 17:57:32.658711 (+ 74us) server_negotiation.cc:299] Negotiation successful
0929 17:57:32.658821 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":285,"thread_start_us":148,"threads_started":1}
I20260929 17:57:32.659584 23109 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40643 } has no permanent_uuid. Determining permanent_uuid...
I20260929 17:57:32.660254 23112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:32.659985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40643 (local address 127.0.0.1:56212)
0929 17:57:32.660094 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0929 17:57:32.660176 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40643: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260929 17:57:32.661026 23112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:32.660795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40643 (local address 127.0.0.1:56222)
0929 17:57:32.660912 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0929 17:57:32.660953 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40643: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260929 17:57:32.661170 23109 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:40643: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40643: connect: Connection refused (error 111)
I20260929 17:57:32.692341 23109 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40643 } attempt: 1
I20260929 17:57:32.693228 23112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:32.692903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40643 (local address 127.0.0.1:56228)
0929 17:57:32.693078 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0929 17:57:32.693145 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40643: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260929 17:57:32.693995 23112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:32.693740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40643 (local address 127.0.0.1:56244)
0929 17:57:32.693842 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0929 17:57:32.693894 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40643: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260929 17:57:32.694135 23109 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:40643: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40643: connect: Connection refused (error 111)
I20260929 17:57:32.708169 23184 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40643 } attempt: 1
I20260929 17:57:32.708992 23186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:32.708705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40643 (local address 127.0.0.1:56258)
0929 17:57:32.708848 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0929 17:57:32.708908 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40643: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260929 17:57:32.709710 23186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:32.709504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40643 (local address 127.0.0.1:56270)
0929 17:57:32.709606 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0929 17:57:32.709652 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40643: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260929 17:57:32.709854 23184 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:40643: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40643: connect: Connection refused (error 111)
W20260929 17:57:32.768106 23191 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 17:57:32.768399 23191 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 17:57:32.768422 23191 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 17:57:32.768482 23191 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260929 17:57:32.769286 23109 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40643 } attempt: 2
I20260929 17:57:32.770073 23112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:32.769799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40643 (local address 127.0.0.1:56278)
0929 17:57:32.769939 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0929 17:57:32.769999 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40643: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260929 17:57:32.770711 23112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:32.770528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40643 (local address 127.0.0.1:56294)
0929 17:57:32.770612 (+ 84us) negotiation.cc:107] Waiting for socket to connect
0929 17:57:32.770657 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40643: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260929 17:57:32.770820 23109 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:40643: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40643: connect: Connection refused (error 111)
W20260929 17:57:32.773113 23191 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 17:57:32.773165 23191 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 17:57:32.773183 23191 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 17:57:32.773203 23191 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 17:57:32.773224 23191 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 17:57:32.778005 23184 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40643 } attempt: 2
I20260929 17:57:32.778816 23186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:32.778545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40643 (local address 127.0.0.1:56306)
0929 17:57:32.778686 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0929 17:57:32.778747 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40643: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260929 17:57:32.778863 23191 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790704375975685-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790704375975685-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44477,127.0.98.61:33593,127.0.98.60:40643
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790704375975685-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:40643
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790704375975685-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23191
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790704375975685-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 16:58:45 UTC on ccycloud.quasar-jpsayz.root.comops.site
build id 4680676
I20260929 17:57:32.779531 23186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:32.779342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40643 (local address 127.0.0.1:56314)
0929 17:57:32.779429 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0929 17:57:32.779473 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40643: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260929 17:57:32.779640 23184 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:40643: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40643: connect: Connection refused (error 111)
I20260929 17:57:32.780089 23191 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 17:57:32.781096 23191 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 17:57:32.789633 23197 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:57:32.789633 23200 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 17:57:32.789675 23198 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 17:57:32.790051 23191 server_base.cc:1034] running on GCE node
I20260929 17:57:32.790673 23191 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 17:57:32.791843 23191 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 17:57:32.793057 23191 hybrid_clock.cc:648] HybridClock initialized: now 1790704652793032 us; error 40 us; skew 500 ppm
Sep 29 17:57:32 dist-test-slave-53nh krb5kdc[23028](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704652, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 17:57:32.795830 23191 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260929 17:57:32.796842 23191 webserver.cc:460] Webserver started at http://127.0.98.60:34331/ using document root <none> and password file <none>
I20260929 17:57:32.797458 23191 fs_manager.cc:359] Metadata directory not provided
I20260929 17:57:32.797524 23191 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 17:57:32.797767 23191 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 17:57:32.799893 23191 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790704375975685-392-0/minicluster-data/master-2/data/instance:
uuid: "d0877b13b3224f6d9c226e390368d69f"
format_stamp: "Formatted at 2026-09-29 17:57:32 on dist-test-slave-53nh"
server_key: "e7219c47299ce6f8577432a038cd82e4"
server_key_iv: "55fdbc6e4872fb5e321cbcb7c963bf31"
server_key_version: "encryptionkey@0"
I20260929 17:57:32.800454 23191 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790704375975685-392-0/minicluster-data/master-2/wal/instance:
uuid: "d0877b13b3224f6d9c226e390368d69f"
format_stamp: "Formatted at 2026-09-29 17:57:32 on dist-test-slave-53nh"
server_key: "e7219c47299ce6f8577432a038cd82e4"
server_key_iv: "55fdbc6e4872fb5e321cbcb7c963bf31"
server_key_version: "encryptionkey@0"
I20260929 17:57:32.804314 23191 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 17:57:32.806756 23207 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 17:57:32.808032 23191 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260929 17:57:32.808180 23191 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790704375975685-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790704375975685-392-0/minicluster-data/master-2/wal
uuid: "d0877b13b3224f6d9c226e390368d69f"
format_stamp: "Formatted at 2026-09-29 17:57:32 on dist-test-slave-53nh"
server_key: "e7219c47299ce6f8577432a038cd82e4"
server_key_iv: "55fdbc6e4872fb5e321cbcb7c963bf31"
server_key_version: "encryptionkey@0"
I20260929 17:57:32.808293 23191 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790704375975685-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790704375975685-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790704375975685-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 17:57:32.829286 23191 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 17:57:32.832393 23191 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 17:57:32.832574 23191 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 17:57:32.839746 23191 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:40643
I20260929 17:57:32.839988 23259 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:40643 every 8 connection(s)
I20260929 17:57:32.840744 23191 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790704375975685-392-0/minicluster-data/master-2/data/info.pb
I20260929 17:57:32.843474 23260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 17:57:32.844269 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskup3Soe/build/debug/bin/kudu as pid 23191
I20260929 17:57:32.844389 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskup3Soe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790704375975685-392-0/minicluster-data/master-2/wal/instance
I20260929 17:57:32.844702 392 external_mini_cluster.cc:1428] Setting key cd0bb66d03b6ccd27d5e188a12e7a8ce
I20260929 17:57:32.847961 23260 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44477 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 17:57:32 dist-test-slave-53nh krb5kdc[23028](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704652, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 29 17:57:32 dist-test-slave-53nh krb5kdc[23028](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704652, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 17:57:32.858525 23263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:32.846031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56324 (local address 127.0.98.60:40643)
0929 17:57:32.846442 (+ 411us) server_negotiation.cc:206] Beginning negotiation
0929 17:57:32.846461 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0929 17:57:32.846506 (+ 45us) server_negotiation.cc:407] Connection header received
0929 17:57:32.847304 (+ 798us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:57:32.847328 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:57:32.847778 (+ 450us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:57:32.848134 (+ 356us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:57:32.848809 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:32.850114 (+ 1305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:32.851005 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:32.851523 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:32.851647 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:32.854234 (+ 2587us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:57:32.854241 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:57:32.854254 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:57:32.854280 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:57:32.855883 (+ 1603us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:57:32.856314 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:57:32.856320 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:57:32.856325 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:57:32.856454 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:57:32.856724 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:57:32.856726 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:57:32.856728 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:57:32.857136 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:57:32.857313 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0929 17:57:32.857512 (+ 199us) server_negotiation.cc:299] Negotiation successful
0929 17:57:32.857803 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":269,"thread_start_us":114,"threads_started":1}
I20260929 17:57:32.858686 23113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:32.849868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59210 (local address 127.0.98.62:44477)
0929 17:57:32.850008 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0929 17:57:32.850012 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:57:32.850293 (+ 281us) server_negotiation.cc:407] Connection header received
0929 17:57:32.850411 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:57:32.850414 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:57:32.850456 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:57:32.850520 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:57:32.851196 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:32.851703 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:32.852700 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:32.853055 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:32.853176 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:32.855538 (+ 2362us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:57:32.855541 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:57:32.855543 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:57:32.855567 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:57:32.856993 (+ 1426us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:57:32.857468 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:57:32.857471 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:57:32.857473 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:57:32.857559 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:57:32.857836 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:57:32.857840 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:57:32.857841 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:57:32.858017 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:57:32.858097 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0929 17:57:32.858428 (+ 331us) server_negotiation.cc:299] Negotiation successful
0929 17:57:32.858529 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 17:57:32.858711 23264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:32.849766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44477 (local address 127.0.0.1:59210)
0929 17:57:32.850170 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0929 17:57:32.850219 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 17:57:32.850328 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:57:32.850576 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:57:32.850581 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:57:32.850917 (+ 336us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:57:32.851055 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:57:32.851062 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:57:32.851817 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:57:32.851820 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:57:32.852581 (+ 761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:57:32.852591 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:57:32.853236 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:57:32.853240 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:57:32.853347 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:32.853826 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:57:32.853844 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:57:32.855409 (+ 1565us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:57:32.857118 (+ 1709us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:57:32.857124 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:57:32.857129 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:57:32.857377 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:57:32.857666 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:57:32.857669 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:57:32.857671 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:57:32.857752 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:57:32.858113 (+ 361us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:57:32.858119 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:57:32.858232 (+ 113us) client_negotiation.cc:769] Sending connection context
0929 17:57:32.858344 (+ 112us) client_negotiation.cc:240] Negotiation successful
0929 17:57:32.858510 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":134,"threads_started":1}
I20260929 17:57:32.859694 23260 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33593 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 17:57:32 dist-test-slave-53nh krb5kdc[23028](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704652, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Sep 29 17:57:32 dist-test-slave-53nh krb5kdc[23028](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704652, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 17:57:32.867853 23264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:32.860102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33593 (local address 127.0.0.1:44742)
0929 17:57:32.860392 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0929 17:57:32.860403 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 17:57:32.860481 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:57:32.860826 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:57:32.860830 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:57:32.860977 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:57:32.861106 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:57:32.861113 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:57:32.861985 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:57:32.861987 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:57:32.862674 (+ 687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:57:32.862684 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:57:32.863167 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:57:32.863169 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:57:32.863237 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:32.863632 (+ 395us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:57:32.863642 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:57:32.864766 (+ 1124us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:57:32.866505 (+ 1739us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:57:32.866507 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:57:32.866509 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:57:32.866685 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:57:32.866943 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:57:32.866946 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:57:32.866947 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:57:32.867031 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:57:32.867414 (+ 383us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:57:32.867417 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:57:32.867512 (+ 95us) client_negotiation.cc:769] Sending connection context
0929 17:57:32.867612 (+ 100us) client_negotiation.cc:240] Negotiation successful
0929 17:57:32.867728 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239}
I20260929 17:57:32.868201 23190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:32.860214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44742 (local address 127.0.98.61:33593)
0929 17:57:32.860327 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0929 17:57:32.860330 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 17:57:32.860455 (+ 125us) server_negotiation.cc:407] Connection header received
0929 17:57:32.860569 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:57:32.860572 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:57:32.860638 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:57:32.860713 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:57:32.861230 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:32.861877 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:32.862806 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:32.863063 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:32.863159 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:32.864896 (+ 1737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:57:32.864900 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:57:32.864903 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:57:32.864931 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:57:32.866410 (+ 1479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:57:32.866774 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:57:32.866777 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:57:32.866779 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:57:32.866864 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:57:32.867111 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:57:32.867113 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:57:32.867114 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:57:32.867314 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:57:32.867408 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0929 17:57:32.867679 (+ 271us) server_negotiation.cc:299] Negotiation successful
0929 17:57:32.867812 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 17:57:32.868633 23113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:32.860663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59222 (local address 127.0.98.62:44477)
0929 17:57:32.860767 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0929 17:57:32.860771 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:57:32.860783 (+ 12us) server_negotiation.cc:407] Connection header received
0929 17:57:32.860888 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:57:32.860890 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:57:32.860921 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:57:32.860975 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:57:32.861575 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:32.862175 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:32.863058 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:32.863350 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:32.863420 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:32.865533 (+ 2113us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:57:32.865537 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:57:32.865539 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:57:32.865563 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:57:32.866857 (+ 1294us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:57:32.867254 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:57:32.867257 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:57:32.867259 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:57:32.867336 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:57:32.867577 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:57:32.867579 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:57:32.867580 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:57:32.867790 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:57:32.868022 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0929 17:57:32.868265 (+ 243us) server_negotiation.cc:299] Negotiation successful
0929 17:57:32.868380 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 17:57:32.868912 23260 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40643 } has no permanent_uuid. Determining permanent_uuid...
I20260929 17:57:32.869951 23109 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40643 } attempt: 3
I20260929 17:57:32.870786 23184 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40643 } attempt: 3
Sep 29 17:57:32 dist-test-slave-53nh krb5kdc[23028](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704652, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 29 17:57:32 dist-test-slave-53nh krb5kdc[23028](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704652, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 29 17:57:32 dist-test-slave-53nh krb5kdc[23028](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704652, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 29 17:57:32 dist-test-slave-53nh krb5kdc[23028](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790704652, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260929 17:57:32.878679 23264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:32.869384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40643 (local address 127.0.0.1:56332)
0929 17:57:32.869485 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0929 17:57:32.869497 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 17:57:32.869563 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:57:32.869846 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:57:32.869849 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:57:32.870008 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:57:32.870089 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:57:32.870095 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:57:32.871372 (+ 1277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:57:32.871374 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:57:32.872101 (+ 727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:57:32.872109 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:57:32.872704 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:57:32.872707 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:57:32.872779 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:32.873916 (+ 1137us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:57:32.873929 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:57:32.875613 (+ 1684us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:57:32.877268 (+ 1655us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:57:32.877271 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:57:32.877273 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:57:32.877516 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:57:32.877767 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:57:32.877769 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:57:32.877771 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:57:32.877843 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:57:32.878208 (+ 365us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:57:32.878211 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:57:32.878293 (+ 82us) client_negotiation.cc:769] Sending connection context
0929 17:57:32.878385 (+ 92us) client_negotiation.cc:240] Negotiation successful
0929 17:57:32.878503 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260929 17:57:32.879220 23263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:32.869504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56332 (local address 127.0.98.60:40643)
0929 17:57:32.869604 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0929 17:57:32.869609 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:57:32.869620 (+ 11us) server_negotiation.cc:407] Connection header received
0929 17:57:32.869646 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:57:32.869649 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:57:32.869694 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:57:32.869757 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:57:32.870198 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:32.871273 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:32.872205 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:32.872513 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:32.872594 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:32.875759 (+ 3165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:57:32.875763 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:57:32.875766 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:57:32.875788 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:57:32.877181 (+ 1393us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:57:32.877622 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:57:32.877625 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:57:32.877627 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:57:32.877689 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:57:32.877928 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:57:32.877930 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:57:32.877932 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:57:32.878125 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:57:32.878208 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0929 17:57:32.878932 (+ 724us) server_negotiation.cc:299] Negotiation successful
0929 17:57:32.879069 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260929 17:57:32.881183 23190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:32.869810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44752 (local address 127.0.98.61:33593)
0929 17:57:32.869912 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0929 17:57:32.869916 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:57:32.869939 (+ 23us) server_negotiation.cc:407] Connection header received
0929 17:57:32.870279 (+ 340us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:57:32.870282 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:57:32.870325 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:57:32.870413 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:57:32.871510 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:32.872308 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:32.873367 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:32.873694 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:32.873757 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:32.877398 (+ 3641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:57:32.877401 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:57:32.877404 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:57:32.877428 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:57:32.879600 (+ 2172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:57:32.880070 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:57:32.880074 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:57:32.880076 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:57:32.880153 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:57:32.880398 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:57:32.880401 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:57:32.880403 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:57:32.880566 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:57:32.880755 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0929 17:57:32.880901 (+ 146us) server_negotiation.cc:299] Negotiation successful
0929 17:57:32.881032 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 17:57:32.881569 23186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:32.871100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40643 (local address 127.0.0.1:56354)
0929 17:57:32.871250 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0929 17:57:32.871263 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 17:57:32.871329 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:57:32.871968 (+ 639us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:57:32.871971 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:57:32.872136 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:57:32.872244 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:57:32.872251 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:57:32.873254 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:57:32.873257 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:57:32.873773 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:57:32.873781 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:57:32.874360 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:57:32.874364 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:57:32.874442 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:32.874837 (+ 395us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:57:32.874851 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:57:32.877610 (+ 2759us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:57:32.880136 (+ 2526us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:57:32.880139 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:57:32.880141 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:57:32.880367 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:57:32.880654 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:57:32.880657 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:57:32.880659 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:57:32.880743 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:57:32.881151 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:57:32.881154 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:57:32.881245 (+ 91us) client_negotiation.cc:769] Sending connection context
0929 17:57:32.881322 (+ 77us) client_negotiation.cc:240] Negotiation successful
0929 17:57:32.881440 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92}
I20260929 17:57:32.882045 23266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:32.871488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56354 (local address 127.0.98.60:40643)
0929 17:57:32.871757 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0929 17:57:32.871765 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 17:57:32.871778 (+ 13us) server_negotiation.cc:407] Connection header received
0929 17:57:32.871807 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:57:32.871810 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:57:32.871845 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:57:32.872335 (+ 490us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:57:32.872443 (+ 108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:32.873169 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:32.873876 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:32.874210 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:32.874347 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:32.877748 (+ 3401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:57:32.877751 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:57:32.877754 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:57:32.877776 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:57:32.880024 (+ 2248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:57:32.880480 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:57:32.880482 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:57:32.880484 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:57:32.880555 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:57:32.880838 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:57:32.880841 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:57:32.880842 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:57:32.881039 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:57:32.881726 (+ 687us) server_negotiation.cc:1035] Waiting for connection context
0929 17:57:32.881796 (+ 70us) server_negotiation.cc:299] Negotiation successful
0929 17:57:32.881895 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":211,"thread_start_us":83,"threads_started":1}
I20260929 17:57:32.882766 23260 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0877b13b3224f6d9c226e390368d69f: Bootstrap starting.
I20260929 17:57:32.884270 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 17:57:32.884814 23184 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac2d4de906974b579e10066188cb24a6: Bootstrap starting.
I20260929 17:57:32.885558 23260 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d0877b13b3224f6d9c226e390368d69f: Neither blocks nor log segments found. Creating new log.
I20260929 17:57:32.886286 23112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:32.870307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40643 (local address 127.0.0.1:56338)
0929 17:57:32.870427 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0929 17:57:32.870440 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 17:57:32.870502 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 17:57:32.871136 (+ 634us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 17:57:32.871139 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 17:57:32.871294 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0929 17:57:32.871383 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:57:32.871389 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:57:32.872475 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:57:32.872478 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:57:32.873592 (+ 1114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 17:57:32.873602 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 17:57:32.874429 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 17:57:32.874433 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 17:57:32.874511 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:32.875139 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 17:57:32.875176 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0929 17:57:32.877223 (+ 2047us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 17:57:32.879679 (+ 2456us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:57:32.879682 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:57:32.879684 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:57:32.879918 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:57:32.885294 (+ 5376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 17:57:32.885298 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 17:57:32.885300 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 17:57:32.885399 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 17:57:32.885832 (+ 433us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 17:57:32.885835 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 17:57:32.885929 (+ 94us) client_negotiation.cc:769] Sending connection context
0929 17:57:32.886005 (+ 76us) client_negotiation.cc:240] Negotiation successful
0929 17:57:32.886139 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260929 17:57:32.886401 23260 log.cc:824] T 00000000000000000000000000000000 P d0877b13b3224f6d9c226e390368d69f: Log is configured to *not* fsync() on all Append() calls
I20260929 17:57:32.885401 23083 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "eb7d936e30da40d2b5b05cb87da5ee13"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59222
I20260929 17:57:32.886790 23265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:32.870539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56338 (local address 127.0.98.60:40643)
0929 17:57:32.870895 (+ 356us) server_negotiation.cc:206] Beginning negotiation
0929 17:57:32.870903 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 17:57:32.870915 (+ 12us) server_negotiation.cc:407] Connection header received
0929 17:57:32.870945 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:57:32.870947 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:57:32.870986 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:57:32.871487 (+ 501us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:57:32.871590 (+ 103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:32.872364 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:32.873720 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:32.874054 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:32.875693 (+ 1639us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:32.877341 (+ 1648us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:57:32.877344 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:57:32.877347 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:57:32.877368 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:57:32.879545 (+ 2177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:57:32.885086 (+ 5541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:57:32.885089 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:57:32.885091 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:57:32.885190 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:57:32.885513 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:57:32.885516 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:57:32.885518 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:57:32.885725 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:57:32.886441 (+ 716us) server_negotiation.cc:1035] Waiting for connection context
0929 17:57:32.886515 (+ 74us) server_negotiation.cc:299] Negotiation successful
0929 17:57:32.886626 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":117,"threads_started":1}
I20260929 17:57:32.887208 23184 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ac2d4de906974b579e10066188cb24a6: Neither blocks nor log segments found. Creating new log.
I20260929 17:57:32.887993 23184 log.cc:824] T 00000000000000000000000000000000 P ac2d4de906974b579e10066188cb24a6: Log is configured to *not* fsync() on all Append() calls
I20260929 17:57:32.888988 23260 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0877b13b3224f6d9c226e390368d69f: No bootstrap required, opened a new log
I20260929 17:57:32.889746 23109 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb7d936e30da40d2b5b05cb87da5ee13: Bootstrap starting.
I20260929 17:57:32.889953 23184 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac2d4de906974b579e10066188cb24a6: No bootstrap required, opened a new log
I20260929 17:57:32.892035 23109 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eb7d936e30da40d2b5b05cb87da5ee13: Neither blocks nor log segments found. Creating new log.
I20260929 17:57:32.892774 23109 log.cc:824] T 00000000000000000000000000000000 P eb7d936e30da40d2b5b05cb87da5ee13: Log is configured to *not* fsync() on all Append() calls
I20260929 17:57:32.893816 23184 raft_consensus.cc:348] T 00000000000000000000000000000000 P ac2d4de906974b579e10066188cb24a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb7d936e30da40d2b5b05cb87da5ee13" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44477 } } peers { permanent_uuid: "ac2d4de906974b579e10066188cb24a6" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33593 } } peers { permanent_uuid: "d0877b13b3224f6d9c226e390368d69f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40643 } }
I20260929 17:57:32.893821 23260 raft_consensus.cc:348] T 00000000000000000000000000000000 P d0877b13b3224f6d9c226e390368d69f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb7d936e30da40d2b5b05cb87da5ee13" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44477 } } peers { permanent_uuid: "ac2d4de906974b579e10066188cb24a6" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33593 } } peers { permanent_uuid: "d0877b13b3224f6d9c226e390368d69f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40643 } }
I20260929 17:57:32.894049 23184 raft_consensus.cc:729] T 00000000000000000000000000000000 P ac2d4de906974b579e10066188cb24a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac2d4de906974b579e10066188cb24a6, State: Initialized, Role: FOLLOWER
I20260929 17:57:32.894100 23260 raft_consensus.cc:729] T 00000000000000000000000000000000 P d0877b13b3224f6d9c226e390368d69f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0877b13b3224f6d9c226e390368d69f, State: Initialized, Role: FOLLOWER
I20260929 17:57:32.894376 23260 consensus_queue.cc:260] T 00000000000000000000000000000000 P d0877b13b3224f6d9c226e390368d69f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb7d936e30da40d2b5b05cb87da5ee13" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44477 } } peers { permanent_uuid: "ac2d4de906974b579e10066188cb24a6" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33593 } } peers { permanent_uuid: "d0877b13b3224f6d9c226e390368d69f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40643 } }
I20260929 17:57:32.894376 23184 consensus_queue.cc:260] T 00000000000000000000000000000000 P ac2d4de906974b579e10066188cb24a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb7d936e30da40d2b5b05cb87da5ee13" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44477 } } peers { permanent_uuid: "ac2d4de906974b579e10066188cb24a6" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33593 } } peers { permanent_uuid: "d0877b13b3224f6d9c226e390368d69f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40643 } }
I20260929 17:57:32.894606 23109 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb7d936e30da40d2b5b05cb87da5ee13: No bootstrap required, opened a new log
I20260929 17:57:32.894951 23270 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac2d4de906974b579e10066188cb24a6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb7d936e30da40d2b5b05cb87da5ee13" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44477 } } peers { permanent_uuid: "ac2d4de906974b579e10066188cb24a6" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33593 } } peers { permanent_uuid: "d0877b13b3224f6d9c226e390368d69f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40643 } } }
I20260929 17:57:32.895254 23270 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac2d4de906974b579e10066188cb24a6 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 17:57:32.895630 23184 sys_catalog.cc:564] T 00000000000000000000000000000000 P ac2d4de906974b579e10066188cb24a6 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 17:57:32.895630 23260 sys_catalog.cc:564] T 00000000000000000000000000000000 P d0877b13b3224f6d9c226e390368d69f [sys.catalog]: configured and running, proceeding with master startup.
I20260929 17:57:32.897410 23109 raft_consensus.cc:348] T 00000000000000000000000000000000 P eb7d936e30da40d2b5b05cb87da5ee13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb7d936e30da40d2b5b05cb87da5ee13" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44477 } } peers { permanent_uuid: "ac2d4de906974b579e10066188cb24a6" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33593 } } peers { permanent_uuid: "d0877b13b3224f6d9c226e390368d69f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40643 } }
I20260929 17:57:32.897616 23109 raft_consensus.cc:729] T 00000000000000000000000000000000 P eb7d936e30da40d2b5b05cb87da5ee13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb7d936e30da40d2b5b05cb87da5ee13, State: Initialized, Role: FOLLOWER
I20260929 17:57:32.897922 23109 consensus_queue.cc:260] T 00000000000000000000000000000000 P eb7d936e30da40d2b5b05cb87da5ee13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb7d936e30da40d2b5b05cb87da5ee13" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44477 } } peers { permanent_uuid: "ac2d4de906974b579e10066188cb24a6" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33593 } } peers { permanent_uuid: "d0877b13b3224f6d9c226e390368d69f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40643 } }
I20260929 17:57:32.898550 23271 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0877b13b3224f6d9c226e390368d69f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb7d936e30da40d2b5b05cb87da5ee13" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44477 } } peers { permanent_uuid: "ac2d4de906974b579e10066188cb24a6" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33593 } } peers { permanent_uuid: "d0877b13b3224f6d9c226e390368d69f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40643 } } }
I20260929 17:57:32.898670 23271 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0877b13b3224f6d9c226e390368d69f [sys.catalog]: This master's current role is: FOLLOWER
W20260929 17:57:32.901243 23290 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d0877b13b3224f6d9c226e390368d69f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 17:57:32.901317 23290 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260929 17:57:32.901257 23293 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ac2d4de906974b579e10066188cb24a6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 17:57:32.901443 23293 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 17:57:32.899240 23191 master_runner.cc:186] Error getting master registration for 127.0.98.62:44477: OK, instance_id { permanent_uuid: "eb7d936e30da40d2b5b05cb87da5ee13" instance_seqno: 1790704652406627 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 17:57:32.902150 23191 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 17:57:32.899919 23116 master_runner.cc:186] Error getting master registration for 127.0.98.62:44477: OK, instance_id { permanent_uuid: "eb7d936e30da40d2b5b05cb87da5ee13" instance_seqno: 1790704652406627 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 17:57:32.902634 23116 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 17:57:32.903599 23109 sys_catalog.cc:564] T 00000000000000000000000000000000 P eb7d936e30da40d2b5b05cb87da5ee13 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 17:57:32.903672 23083 raft_consensus.cc:482] T 00000000000000000000000000000000 P eb7d936e30da40d2b5b05cb87da5ee13 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260929 17:57:32.903764 23083 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eb7d936e30da40d2b5b05cb87da5ee13 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:57:32.905068 23083 raft_consensus.cc:504] T 00000000000000000000000000000000 P eb7d936e30da40d2b5b05cb87da5ee13 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb7d936e30da40d2b5b05cb87da5ee13" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44477 } } peers { permanent_uuid: "ac2d4de906974b579e10066188cb24a6" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33593 } } peers { permanent_uuid: "d0877b13b3224f6d9c226e390368d69f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40643 } }
I20260929 17:57:32.906039 23083 leader_election.cc:288] T 00000000000000000000000000000000 P eb7d936e30da40d2b5b05cb87da5ee13 [CANDIDATE]: Term 1 election: Requested vote from peers ac2d4de906974b579e10066188cb24a6 (127.0.98.61:33593), d0877b13b3224f6d9c226e390368d69f (127.0.98.60:40643)
I20260929 17:57:32.906839 23234 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "eb7d936e30da40d2b5b05cb87da5ee13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "d0877b13b3224f6d9c226e390368d69f"
I20260929 17:57:32.907079 23234 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d0877b13b3224f6d9c226e390368d69f [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:57:32.907251 23158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "eb7d936e30da40d2b5b05cb87da5ee13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "ac2d4de906974b579e10066188cb24a6"
I20260929 17:57:32.907480 23158 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ac2d4de906974b579e10066188cb24a6 [term 0 FOLLOWER]: Advancing to term 1
I20260929 17:57:32.908444 23234 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d0877b13b3224f6d9c226e390368d69f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb7d936e30da40d2b5b05cb87da5ee13 in term 1.
I20260929 17:57:32.908756 23158 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ac2d4de906974b579e10066188cb24a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb7d936e30da40d2b5b05cb87da5ee13 in term 1.
W20260929 17:57:32.908762 23307 catalog_manager.cc:1558] T 00000000000000000000000000000000 P eb7d936e30da40d2b5b05cb87da5ee13: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 17:57:32.908833 23307 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 17:57:32.909287 23060 leader_election.cc:302] T 00000000000000000000000000000000 P eb7d936e30da40d2b5b05cb87da5ee13 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac2d4de906974b579e10066188cb24a6, eb7d936e30da40d2b5b05cb87da5ee13; no voters:
I20260929 17:57:32.909615 23294 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb7d936e30da40d2b5b05cb87da5ee13 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb7d936e30da40d2b5b05cb87da5ee13" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44477 } } peers { permanent_uuid: "ac2d4de906974b579e10066188cb24a6" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33593 } } peers { permanent_uuid: "d0877b13b3224f6d9c226e390368d69f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40643 } } }
I20260929 17:57:32.911139 23312 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eb7d936e30da40d2b5b05cb87da5ee13 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 17:57:32.911387 23294 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb7d936e30da40d2b5b05cb87da5ee13 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 17:57:32.911525 23312 raft_consensus.cc:686] T 00000000000000000000000000000000 P eb7d936e30da40d2b5b05cb87da5ee13 [term 1 LEADER]: Becoming Leader. State: Replica: eb7d936e30da40d2b5b05cb87da5ee13, State: Running, Role: LEADER
I20260929 17:57:32.912480 23312 consensus_queue.cc:237] T 00000000000000000000000000000000 P eb7d936e30da40d2b5b05cb87da5ee13 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb7d936e30da40d2b5b05cb87da5ee13" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44477 } } peers { permanent_uuid: "ac2d4de906974b579e10066188cb24a6" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33593 } } peers { permanent_uuid: "d0877b13b3224f6d9c226e390368d69f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40643 } }
I20260929 17:57:32.914980 23041 master_runner.cc:263] No leader master found from master eb7d936e30da40d2b5b05cb87da5ee13
I20260929 17:57:32.915048 23041 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 17:57:32.915414 23294 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb7d936e30da40d2b5b05cb87da5ee13 [sys.catalog]: SysCatalogTable state changed. Reason: New leader eb7d936e30da40d2b5b05cb87da5ee13. Latest consensus state: current_term: 1 leader_uuid: "eb7d936e30da40d2b5b05cb87da5ee13" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb7d936e30da40d2b5b05cb87da5ee13" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44477 } } peers { permanent_uuid: "ac2d4de906974b579e10066188cb24a6" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33593 } } peers { permanent_uuid: "d0877b13b3224f6d9c226e390368d69f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40643 } } }
I20260929 17:57:32.915563 23294 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb7d936e30da40d2b5b05cb87da5ee13 [sys.catalog]: This master's current role is: LEADER
I20260929 17:57:32.923523 23319 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 17:57:32.924360 23190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:32.912024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44754 (local address 127.0.98.61:33593)
0929 17:57:32.912139 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0929 17:57:32.912143 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:57:32.915670 (+ 3527us) server_negotiation.cc:407] Connection header received
0929 17:57:32.915871 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:57:32.915875 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:57:32.915932 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:57:32.915985 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:57:32.916946 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:32.917693 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:32.918783 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:32.919109 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:32.919191 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:32.920926 (+ 1735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:57:32.920930 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:57:32.920933 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:57:32.920961 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:57:32.922591 (+ 1630us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:57:32.923044 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:57:32.923048 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:57:32.923050 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:57:32.923127 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:57:32.923437 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:57:32.923440 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:57:32.923442 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:57:32.923651 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:57:32.923747 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0929 17:57:32.924074 (+ 327us) server_negotiation.cc:299] Negotiation successful
0929 17:57:32.924212 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 17:57:32.924604 23319 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 17:57:32.933038 23234 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d0877b13b3224f6d9c226e390368d69f [term 1 FOLLOWER]: Refusing update from remote peer eb7d936e30da40d2b5b05cb87da5ee13: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 17:57:32.933696 23265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:32.914934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56368 (local address 127.0.98.60:40643)
0929 17:57:32.915041 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0929 17:57:32.915044 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 17:57:32.918733 (+ 3689us) server_negotiation.cc:407] Connection header received
0929 17:57:32.920631 (+ 1898us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:57:32.920634 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:57:32.920703 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:57:32.920754 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:57:32.921523 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:32.922289 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:32.927530 (+ 5241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:32.927916 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:32.927997 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:32.929580 (+ 1583us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:57:32.929584 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:57:32.929587 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:57:32.929619 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:57:32.931129 (+ 1510us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:57:32.932354 (+ 1225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:57:32.932357 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:57:32.932359 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:57:32.932448 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:57:32.932746 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:57:32.932749 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:57:32.932751 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:57:32.932951 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:57:32.933096 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0929 17:57:32.933419 (+ 323us) server_negotiation.cc:299] Negotiation successful
0929 17:57:32.933538 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260929 17:57:32.934446 23294 consensus_queue.cc:1035] T 00000000000000000000000000000000 P eb7d936e30da40d2b5b05cb87da5ee13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0877b13b3224f6d9c226e390368d69f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 17:57:32.938314 23271 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0877b13b3224f6d9c226e390368d69f [sys.catalog]: SysCatalogTable state changed. Reason: New leader eb7d936e30da40d2b5b05cb87da5ee13. Latest consensus state: current_term: 1 leader_uuid: "eb7d936e30da40d2b5b05cb87da5ee13" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb7d936e30da40d2b5b05cb87da5ee13" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44477 } } peers { permanent_uuid: "ac2d4de906974b579e10066188cb24a6" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33593 } } peers { permanent_uuid: "d0877b13b3224f6d9c226e390368d69f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40643 } } }
I20260929 17:57:32.938470 23271 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0877b13b3224f6d9c226e390368d69f [sys.catalog]: This master's current role is: FOLLOWER
I20260929 17:57:32.939622 23113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:32.915220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59230 (local address 127.0.98.62:44477)
0929 17:57:32.915309 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0929 17:57:32.915313 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:57:32.917499 (+ 2186us) server_negotiation.cc:407] Connection header received
0929 17:57:32.917642 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:57:32.917645 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:57:32.917699 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:57:32.917745 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:57:32.918799 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:32.919458 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:32.926680 (+ 7222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:32.926986 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:32.927060 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:32.931235 (+ 4175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 17:57:32.931239 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 17:57:32.931241 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 17:57:32.931265 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 17:57:32.934318 (+ 3053us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:57:32.938295 (+ 3977us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:57:32.938299 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:57:32.938300 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:57:32.938391 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 17:57:32.938692 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 17:57:32.938696 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 17:57:32.938697 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 17:57:32.938903 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 17:57:32.939006 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0929 17:57:32.939341 (+ 335us) server_negotiation.cc:299] Negotiation successful
0929 17:57:32.939467 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260929 17:57:32.941087 23158 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ac2d4de906974b579e10066188cb24a6 [term 1 FOLLOWER]: Refusing update from remote peer eb7d936e30da40d2b5b05cb87da5ee13: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 17:57:32.941694 23294 consensus_queue.cc:1035] T 00000000000000000000000000000000 P eb7d936e30da40d2b5b05cb87da5ee13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac2d4de906974b579e10066188cb24a6" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33593 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 17:57:32.944142 23271 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0877b13b3224f6d9c226e390368d69f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "eb7d936e30da40d2b5b05cb87da5ee13" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb7d936e30da40d2b5b05cb87da5ee13" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44477 } } peers { permanent_uuid: "ac2d4de906974b579e10066188cb24a6" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33593 } } peers { permanent_uuid: "d0877b13b3224f6d9c226e390368d69f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40643 } } }
I20260929 17:57:32.944257 23271 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0877b13b3224f6d9c226e390368d69f [sys.catalog]: This master's current role is: FOLLOWER
I20260929 17:57:32.947140 23270 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac2d4de906974b579e10066188cb24a6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader eb7d936e30da40d2b5b05cb87da5ee13. Latest consensus state: current_term: 1 leader_uuid: "eb7d936e30da40d2b5b05cb87da5ee13" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb7d936e30da40d2b5b05cb87da5ee13" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44477 } } peers { permanent_uuid: "ac2d4de906974b579e10066188cb24a6" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33593 } } peers { permanent_uuid: "d0877b13b3224f6d9c226e390368d69f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40643 } } }
I20260929 17:57:32.947296 23270 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac2d4de906974b579e10066188cb24a6 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 17:57:32.947952 23294 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb7d936e30da40d2b5b05cb87da5ee13 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "eb7d936e30da40d2b5b05cb87da5ee13" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb7d936e30da40d2b5b05cb87da5ee13" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44477 } } peers { permanent_uuid: "ac2d4de906974b579e10066188cb24a6" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33593 } } peers { permanent_uuid: "d0877b13b3224f6d9c226e390368d69f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40643 } } }
I20260929 17:57:32.948084 23294 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb7d936e30da40d2b5b05cb87da5ee13 [sys.catalog]: This master's current role is: LEADER
I20260929 17:57:32.948681 23320 mvcc.cc:205] Tried to move back new op lower bound from 7334726258403766272 to 7334726258333564928. Current Snapshot: MvccSnapshot[applied={T|T < 7334726258403766272}]
I20260929 17:57:32.950340 23319 catalog_manager.cc:1347] Generated new cluster ID: 4482a850d5ad4cfeb18c015a4257ee26
I20260929 17:57:32.950399 23319 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 17:57:32.952492 23312 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb7d936e30da40d2b5b05cb87da5ee13 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "eb7d936e30da40d2b5b05cb87da5ee13" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb7d936e30da40d2b5b05cb87da5ee13" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44477 } } peers { permanent_uuid: "ac2d4de906974b579e10066188cb24a6" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33593 } } peers { permanent_uuid: "d0877b13b3224f6d9c226e390368d69f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40643 } } }
I20260929 17:57:32.952605 23312 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb7d936e30da40d2b5b05cb87da5ee13 [sys.catalog]: This master's current role is: LEADER
I20260929 17:57:32.953748 23270 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac2d4de906974b579e10066188cb24a6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "eb7d936e30da40d2b5b05cb87da5ee13" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb7d936e30da40d2b5b05cb87da5ee13" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44477 } } peers { permanent_uuid: "ac2d4de906974b579e10066188cb24a6" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33593 } } peers { permanent_uuid: "d0877b13b3224f6d9c226e390368d69f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40643 } } }
I20260929 17:57:32.953869 23270 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac2d4de906974b579e10066188cb24a6 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 17:57:32.955363 23325 mvcc.cc:205] Tried to move back new op lower bound from 7334726258403766272 to 7334726258333564928. Current Snapshot: MvccSnapshot[applied={T|T < 7334726258403766272}]
I20260929 17:57:32.989634 23319 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 17:57:32.991078 23319 catalog_manager.cc:1504] Loading token signing keys...
I20260929 17:57:33.013286 23319 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eb7d936e30da40d2b5b05cb87da5ee13: Generated new TSK 0
I20260929 17:57:33.014295 23319 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 17:57:33.054953 23190 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:57:33.052336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44758 (local address 127.0.98.61:33593)
0929 17:57:33.052468 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0929 17:57:33.052472 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:57:33.053819 (+ 1347us) server_negotiation.cc:407] Connection header received
0929 17:57:33.054018 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:57:33.054021 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:57:33.054076 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:57:33.054130 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:57:33.054816 (+ 686us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44758: BlockingRecv error: recv got EOF from 127.0.0.1:44758 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260929 17:57:33.056378 23265 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:57:33.053190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56382 (local address 127.0.98.60:40643)
0929 17:57:33.053301 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0929 17:57:33.053305 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:57:33.055446 (+ 2141us) server_negotiation.cc:407] Connection header received
0929 17:57:33.055650 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:57:33.055654 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:57:33.055705 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:57:33.055754 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:57:33.056261 (+ 507us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56382: BlockingRecv error: recv got EOF from 127.0.0.1:56382 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260929 17:57:33.058298 23113 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:57:33.056855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59242 (local address 127.0.98.62:44477)
0929 17:57:33.056967 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0929 17:57:33.056971 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:57:33.057439 (+ 468us) server_negotiation.cc:407] Connection header received
0929 17:57:33.057607 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:57:33.057610 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:57:33.057656 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:57:33.057706 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:57:33.058186 (+ 480us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59242: BlockingRecv error: recv got EOF from 127.0.0.1:59242 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260929 17:57:33.062644 23190 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:57:33.061091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44762 (local address 127.0.98.61:33593)
0929 17:57:33.061200 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0929 17:57:33.061205 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:57:33.061694 (+ 489us) server_negotiation.cc:407] Connection header received
0929 17:57:33.061892 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:57:33.061895 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:57:33.061969 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:57:33.062019 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:57:33.062502 (+ 483us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44762: BlockingRecv error: recv got EOF from 127.0.0.1:44762 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260929 17:57:33.067456 23190 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:57:33.065920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44772 (local address 127.0.98.61:33593)
0929 17:57:33.066020 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0929 17:57:33.066024 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:57:33.066551 (+ 527us) server_negotiation.cc:407] Connection header received
0929 17:57:33.066743 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:57:33.066746 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:57:33.066797 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:57:33.066849 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:57:33.067350 (+ 501us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44772: BlockingRecv error: recv got EOF from 127.0.0.1:44772 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260929 17:57:33.073442 23190 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:57:33.071648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44774 (local address 127.0.98.61:33593)
0929 17:57:33.071769 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0929 17:57:33.071773 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:57:33.072364 (+ 591us) server_negotiation.cc:407] Connection header received
0929 17:57:33.072596 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:57:33.072600 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:57:33.072660 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:57:33.072716 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:57:33.073330 (+ 614us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44774: BlockingRecv error: recv got EOF from 127.0.0.1:44774 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260929 17:57:33.081389 23190 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:57:33.079671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44782 (local address 127.0.98.61:33593)
0929 17:57:33.079820 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0929 17:57:33.079825 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:57:33.080389 (+ 564us) server_negotiation.cc:407] Connection header received
0929 17:57:33.080603 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:57:33.080606 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:57:33.080661 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:57:33.080714 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:57:33.081272 (+ 558us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44782: BlockingRecv error: recv got EOF from 127.0.0.1:44782 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
W20260929 17:57:33.093321 23190 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:57:33.091619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44798 (local address 127.0.98.61:33593)
0929 17:57:33.091738 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0929 17:57:33.091742 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:57:33.092298 (+ 556us) server_negotiation.cc:407] Connection header received
0929 17:57:33.092497 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:57:33.092501 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:57:33.092576 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:57:33.092645 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:57:33.093207 (+ 562us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44798: BlockingRecv error: recv got EOF from 127.0.0.1:44798 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260929 17:57:33.114051 23190 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:57:33.111866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44806 (local address 127.0.98.61:33593)
0929 17:57:33.112653 (+ 787us) server_negotiation.cc:206] Beginning negotiation
0929 17:57:33.112658 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:57:33.112707 (+ 49us) server_negotiation.cc:407] Connection header received
0929 17:57:33.112798 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:57:33.112801 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:57:33.113080 (+ 279us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:57:33.113218 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:57:33.113924 (+ 706us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44806: BlockingRecv error: recv got EOF from 127.0.0.1:44806 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260929 17:57:33.152041 23190 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:57:33.150805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44808 (local address 127.0.98.61:33593)
0929 17:57:33.150937 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0929 17:57:33.150942 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:57:33.151120 (+ 178us) server_negotiation.cc:407] Connection header received
0929 17:57:33.151282 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:57:33.151285 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:57:33.151350 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:57:33.151451 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:57:33.151906 (+ 455us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44808: BlockingRecv error: recv got EOF from 127.0.0.1:44808 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
W20260929 17:57:33.219051 23190 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:57:33.217970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44822 (local address 127.0.98.61:33593)
0929 17:57:33.218135 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0929 17:57:33.218141 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 17:57:33.218232 (+ 91us) server_negotiation.cc:407] Connection header received
0929 17:57:33.218384 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:57:33.218386 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:57:33.218433 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:57:33.218501 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:57:33.218953 (+ 452us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44822: BlockingRecv error: recv got EOF from 127.0.0.1:44822 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260929 17:57:33.350545 23190 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:57:33.349323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44824 (local address 127.0.98.61:33593)
0929 17:57:33.349473 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0929 17:57:33.349477 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:57:33.349611 (+ 134us) server_negotiation.cc:407] Connection header received
0929 17:57:33.349747 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:57:33.349749 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:57:33.349798 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:57:33.349872 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:57:33.350413 (+ 541us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44824: BlockingRecv error: recv got EOF from 127.0.0.1:44824 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
W20260929 17:57:33.609747 23190 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 17:57:33.608625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44826 (local address 127.0.98.61:33593)
0929 17:57:33.608787 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0929 17:57:33.608793 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 17:57:33.608810 (+ 17us) server_negotiation.cc:407] Connection header received
0929 17:57:33.608961 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:57:33.608965 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:57:33.609049 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:57:33.609127 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 17:57:33.609634 (+ 507us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44826: BlockingRecv error: recv got EOF from 127.0.0.1:44826 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
I20260929 17:57:33.902626 23290 catalog_manager.cc:1259] Loaded cluster ID: 4482a850d5ad4cfeb18c015a4257ee26
I20260929 17:57:33.902711 23290 catalog_manager.cc:1552] T 00000000000000000000000000000000 P d0877b13b3224f6d9c226e390368d69f: loading cluster ID for follower catalog manager: success
I20260929 17:57:33.902722 23293 catalog_manager.cc:1259] Loaded cluster ID: 4482a850d5ad4cfeb18c015a4257ee26
I20260929 17:57:33.902788 23293 catalog_manager.cc:1552] T 00000000000000000000000000000000 P ac2d4de906974b579e10066188cb24a6: loading cluster ID for follower catalog manager: success
I20260929 17:57:33.904527 23293 catalog_manager.cc:1574] T 00000000000000000000000000000000 P ac2d4de906974b579e10066188cb24a6: acquiring CA information for follower catalog manager: success
I20260929 17:57:33.904531 23290 catalog_manager.cc:1574] T 00000000000000000000000000000000 P d0877b13b3224f6d9c226e390368d69f: acquiring CA information for follower catalog manager: success
I20260929 17:57:33.905709 23293 catalog_manager.cc:1602] T 00000000000000000000000000000000 P ac2d4de906974b579e10066188cb24a6: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 17:57:33.905709 23290 catalog_manager.cc:1602] T 00000000000000000000000000000000 P d0877b13b3224f6d9c226e390368d69f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 17:57:34.127422 23391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:34.123772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44836 (local address 127.0.98.61:33593)
0929 17:57:34.124020 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0929 17:57:34.124023 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 17:57:34.124081 (+ 58us) server_negotiation.cc:407] Connection header received
0929 17:57:34.124212 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:57:34.124215 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:57:34.124338 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:57:34.124434 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 17:57:34.124634 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:34.125191 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:34.126034 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:34.126265 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:34.126342 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:34.126588 (+ 246us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 17:57:34.127026 (+ 438us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 17:57:34.127116 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0929 17:57:34.127243 (+ 127us) server_negotiation.cc:299] Negotiation successful
0929 17:57:34.127296 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":84,"threads_started":1}
I20260929 17:57:34.135773 23397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:34.130630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56388 (local address 127.0.98.60:40643)
0929 17:57:34.130845 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0929 17:57:34.130849 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:57:34.130914 (+ 65us) server_negotiation.cc:407] Connection header received
0929 17:57:34.131114 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:57:34.131118 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:57:34.131279 (+ 161us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:57:34.131398 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 17:57:34.131624 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:34.132393 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:34.133631 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:34.133985 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:34.134104 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:34.134538 (+ 434us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 17:57:34.135115 (+ 577us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 17:57:34.135229 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0929 17:57:34.135541 (+ 312us) server_negotiation.cc:299] Negotiation successful
0929 17:57:34.135605 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":75,"threads_started":1}
I20260929 17:57:34.148407 23391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:34.144167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44852 (local address 127.0.98.61:33593)
0929 17:57:34.144289 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0929 17:57:34.144294 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:57:34.144309 (+ 15us) server_negotiation.cc:407] Connection header received
0929 17:57:34.144363 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:57:34.144366 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:57:34.144418 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:57:34.144491 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 17:57:34.144785 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:34.145628 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:34.146956 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:34.147296 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:34.147371 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:34.147771 (+ 400us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 17:57:34.147869 (+ 98us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 17:57:34.147916 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0929 17:57:34.148095 (+ 179us) server_negotiation.cc:299] Negotiation successful
0929 17:57:34.148164 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260929 17:57:34.148474 23397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:34.144014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56394 (local address 127.0.98.60:40643)
0929 17:57:34.144137 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0929 17:57:34.144142 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 17:57:34.144726 (+ 584us) server_negotiation.cc:407] Connection header received
0929 17:57:34.144839 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:57:34.144842 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:57:34.144891 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:57:34.144934 (+ 43us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 17:57:34.145299 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:34.146066 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:34.147266 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:34.147584 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:34.147652 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:34.148023 (+ 371us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 17:57:34.148108 (+ 85us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 17:57:34.148153 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0929 17:57:34.148297 (+ 144us) server_negotiation.cc:299] Negotiation successful
0929 17:57:34.148351 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260929 17:57:34.149605 23404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 17:57:34.143863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59256 (local address 127.0.98.62:44477)
0929 17:57:34.144132 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0929 17:57:34.144136 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 17:57:34.144404 (+ 268us) server_negotiation.cc:407] Connection header received
0929 17:57:34.144827 (+ 423us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 17:57:34.144830 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 17:57:34.145078 (+ 248us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 17:57:34.145189 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 17:57:34.145514 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:34.146279 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:34.147365 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 17:57:34.147687 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 17:57:34.147791 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 17:57:34.148173 (+ 382us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 17:57:34.148869 (+ 696us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 17:57:34.148959 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0929 17:57:34.149228 (+ 269us) server_negotiation.cc:299] Negotiation successful
0929 17:57:34.149436 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":184,"thread_start_us":81,"threads_started":1}
I20260929 17:57:34.153402 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskup3Soe/build/debug/bin/kudu with pid 23041
I20260929 17:57:34.158989 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskup3Soe/build/debug/bin/kudu with pid 23116
I20260929 17:57:34.163990 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskup3Soe/build/debug/bin/kudu with pid 23191
2026-09-29T17:57:34Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (3020 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (6053 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (278180 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST